[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 00:57:50 dist-test-slave-dhph krb5kdc[24527](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 00:57:50 dist-test-slave-dhph krb5kdc[24527](info): set up 2 sockets
Aug 06 00:57:50 dist-test-slave-dhph krb5kdc[24527](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 00:57:51.255753 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.242s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 00:57:51 dist-test-slave-dhph krb5kdc[24527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T00:57:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T00:57:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 00:57:51.427162 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:33117
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:41015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:33117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:57:51.527473 24542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:57:51.527709 24542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:57:51.527736 24542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:57:51.529016 24542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 00:57:51.529075 24542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:57:51.529091 24542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:57:51.529105 24542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 00:57:51.529117 24542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 00:57:51.530725 24542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:33117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:33117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:57:51.531004 24542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:57:51.531278 24542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:57:51.539692 24547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:57:51.539692 24548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:57:51.541061 24550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:57:51.541302 24542 server_base.cc:1034] running on GCE node
I20260806 00:57:51.541616 24542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:57:51.541971 24542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:57:51.543180 24542 hybrid_clock.cc:648] HybridClock initialized: now 1785977871543155 us; error 115 us; skew 500 ppm
Aug 06 00:57:51 dist-test-slave-dhph krb5kdc[24527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977871, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:57:51.547350 24542 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 00:57:51.547883 24542 webserver.cc:460] Webserver started at http://127.23.242.62:44791/ using document root <none> and password file <none>
I20260806 00:57:51.548171 24542 fs_manager.cc:359] Metadata directory not provided
I20260806 00:57:51.548231 24542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:57:51.548367 24542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:57:51.549515 24542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "d3927f6b391e468ab4b66f0ee1cd40bf"
format_stamp: "Formatted at 2026-08-06 00:57:51 on dist-test-slave-dhph"
server_key: "ef9714010308ec6522b86d44a805243e"
server_key_iv: "446a10b44404ecc4480e8ac78dc7d75c"
server_key_version: "encryptionkey@0"
I20260806 00:57:51.549966 24542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "d3927f6b391e468ab4b66f0ee1cd40bf"
format_stamp: "Formatted at 2026-08-06 00:57:51 on dist-test-slave-dhph"
server_key: "ef9714010308ec6522b86d44a805243e"
server_key_iv: "446a10b44404ecc4480e8ac78dc7d75c"
server_key_version: "encryptionkey@0"
I20260806 00:57:51.551797 24542 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 00:57:51.553033 24558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:51.553390 24542 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:57:51.553551 24542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "d3927f6b391e468ab4b66f0ee1cd40bf"
format_stamp: "Formatted at 2026-08-06 00:57:51 on dist-test-slave-dhph"
server_key: "ef9714010308ec6522b86d44a805243e"
server_key_iv: "446a10b44404ecc4480e8ac78dc7d75c"
server_key_version: "encryptionkey@0"
I20260806 00:57:51.553674 24542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:57:51.567870 24542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:57:51.570703 24542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:57:51.570909 24542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:57:51.577407 24542 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:33117
I20260806 00:57:51.577445 24614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:33117 every 8 connection(s)
I20260806 00:57:51.577877 24542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 00:57:51.579028 24615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:57:51.581848 24615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3927f6b391e468ab4b66f0ee1cd40bf: Bootstrap starting.
I20260806 00:57:51.582698 24615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3927f6b391e468ab4b66f0ee1cd40bf: Neither blocks nor log segments found. Creating new log.
I20260806 00:57:51.583060 24615 log.cc:824] T 00000000000000000000000000000000 P d3927f6b391e468ab4b66f0ee1cd40bf: Log is configured to *not* fsync() on all Append() calls
I20260806 00:57:51.584244 24615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3927f6b391e468ab4b66f0ee1cd40bf: No bootstrap required, opened a new log
I20260806 00:57:51.584520 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 24542
I20260806 00:57:51.584692 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 00:57:51.585340 24520 external_mini_cluster.cc:1428] Setting key c5bd3e2b2922c64f0892476e822f0e14
I20260806 00:57:51.587009 24615 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3927f6b391e468ab4b66f0ee1cd40bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3927f6b391e468ab4b66f0ee1cd40bf" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 33117 } }
I20260806 00:57:51.587245 24615 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3927f6b391e468ab4b66f0ee1cd40bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:57:51.587289 24615 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3927f6b391e468ab4b66f0ee1cd40bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3927f6b391e468ab4b66f0ee1cd40bf, State: Initialized, Role: FOLLOWER
I20260806 00:57:51.587431 24615 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3927f6b391e468ab4b66f0ee1cd40bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3927f6b391e468ab4b66f0ee1cd40bf" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 33117 } }
I20260806 00:57:51.587522 24615 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3927f6b391e468ab4b66f0ee1cd40bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 00:57:51.587577 24615 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3927f6b391e468ab4b66f0ee1cd40bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 00:57:51.587638 24615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3927f6b391e468ab4b66f0ee1cd40bf [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:57:51.589077 24615 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3927f6b391e468ab4b66f0ee1cd40bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3927f6b391e468ab4b66f0ee1cd40bf" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 33117 } }
I20260806 00:57:51.589224 24615 leader_election.cc:302] T 00000000000000000000000000000000 P d3927f6b391e468ab4b66f0ee1cd40bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3927f6b391e468ab4b66f0ee1cd40bf; no voters:
I20260806 00:57:51.589430 24615 leader_election.cc:288] T 00000000000000000000000000000000 P d3927f6b391e468ab4b66f0ee1cd40bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 00:57:51.589550 24620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3927f6b391e468ab4b66f0ee1cd40bf [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:57:51.589762 24615 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3927f6b391e468ab4b66f0ee1cd40bf [sys.catalog]: configured and running, proceeding with master startup.
I20260806 00:57:51.589977 24620 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3927f6b391e468ab4b66f0ee1cd40bf [term 1 LEADER]: Becoming Leader. State: Replica: d3927f6b391e468ab4b66f0ee1cd40bf, State: Running, Role: LEADER
I20260806 00:57:51.590101 24620 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3927f6b391e468ab4b66f0ee1cd40bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3927f6b391e468ab4b66f0ee1cd40bf" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 33117 } }
I20260806 00:57:51.590529 24621 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3927f6b391e468ab4b66f0ee1cd40bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3927f6b391e468ab4b66f0ee1cd40bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3927f6b391e468ab4b66f0ee1cd40bf" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 33117 } } }
I20260806 00:57:51.590660 24621 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3927f6b391e468ab4b66f0ee1cd40bf [sys.catalog]: This master's current role is: LEADER
I20260806 00:57:51.590832 24622 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3927f6b391e468ab4b66f0ee1cd40bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3927f6b391e468ab4b66f0ee1cd40bf. Latest consensus state: current_term: 1 leader_uuid: "d3927f6b391e468ab4b66f0ee1cd40bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3927f6b391e468ab4b66f0ee1cd40bf" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 33117 } } }
I20260806 00:57:51.590920 24622 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3927f6b391e468ab4b66f0ee1cd40bf [sys.catalog]: This master's current role is: LEADER
I20260806 00:57:51.591094 24629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 00:57:51.591673 24629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 00:57:51.593696 24629 catalog_manager.cc:1347] Generated new cluster ID: 001e3a13df4f4b2aaf4364631047b52f
I20260806 00:57:51.593770 24629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 00:57:51 dist-test-slave-dhph krb5kdc[24527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:57:51.605321 24629 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 00:57:51.606034 24629 catalog_manager.cc:1504] Loading token signing keys...
I20260806 00:57:51.608547 24619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:51.586364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46636 (local address 127.23.242.62:33117)
0806 00:57:51.586610 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:51.586620 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:51.586831 (+ 211us) server_negotiation.cc:407] Connection header received
0806 00:57:51.590280 (+ 3449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:51.590288 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:51.590356 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:51.591532 (+ 1176us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:51.592950 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:51.594457 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:51.596116 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:51.596869 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:51.596999 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:51.602781 (+ 5782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:51.602787 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:51.602788 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:51.602803 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:51.605649 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:51.606322 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:51.606325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:51.606326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:51.606449 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:51.606842 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:51.606843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:51.606844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:51.607160 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:51.607334 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:51.607971 (+ 637us) server_negotiation.cc:299] Negotiation successful
0806 00:57:51.608147 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":86,"threads_started":1}
I20260806 00:57:51.609174 24629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3927f6b391e468ab4b66f0ee1cd40bf: Generated new TSK 0
I20260806 00:57:51.609365 24629 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.242.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
I20260806 00:57:51.661617 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:0
--local_ip_for_outbound_sockets=127.23.242.1
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:33117
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:41015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:57:51.767045 24642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:57:51.767310 24642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:57:51.767355 24642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:57:51.768630 24642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:57:51.768712 24642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:57:51.768764 24642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 00:57:51.770367 24642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:33117
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:57:51.770700 24642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:57:51.770957 24642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:57:51.780205 24651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:57:51.780267 24642 server_base.cc:1034] running on GCE node
W20260806 00:57:51.780225 24649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:57:51.780224 24648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:57:51.780905 24642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:57:51.781193 24642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:57:51.782411 24642 hybrid_clock.cc:648] HybridClock initialized: now 1785977871782369 us; error 96 us; skew 500 ppm
Aug 06 00:57:51 dist-test-slave-dhph krb5kdc[24527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977871, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:57:51.785809 24642 init.cc:377] Logged in from keytab as kudu/127.23.242.1@KRBTEST.COM (short username kudu)
I20260806 00:57:51.786350 24642 webserver.cc:460] Webserver started at http://127.23.242.1:35049/ using document root <none> and password file <none>
I20260806 00:57:51.786717 24642 fs_manager.cc:359] Metadata directory not provided
I20260806 00:57:51.786801 24642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:57:51.786957 24642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:57:51.788156 24642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-0/data/instance:
uuid: "71ba6d56a343410daeb0eded35a00871"
format_stamp: "Formatted at 2026-08-06 00:57:51 on dist-test-slave-dhph"
server_key: "88ec48d7e9b96ed8f6f53c463a8552b9"
server_key_iv: "0479b1494892e869298e37750064188c"
server_key_version: "encryptionkey@0"
I20260806 00:57:51.788583 24642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance:
uuid: "71ba6d56a343410daeb0eded35a00871"
format_stamp: "Formatted at 2026-08-06 00:57:51 on dist-test-slave-dhph"
server_key: "88ec48d7e9b96ed8f6f53c463a8552b9"
server_key_iv: "0479b1494892e869298e37750064188c"
server_key_version: "encryptionkey@0"
I20260806 00:57:51.790529 24642 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260806 00:57:51.791733 24659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:51.792146 24642 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:57:51.792274 24642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "71ba6d56a343410daeb0eded35a00871"
format_stamp: "Formatted at 2026-08-06 00:57:51 on dist-test-slave-dhph"
server_key: "88ec48d7e9b96ed8f6f53c463a8552b9"
server_key_iv: "0479b1494892e869298e37750064188c"
server_key_version: "encryptionkey@0"
I20260806 00:57:51.792364 24642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:57:51.810190 24642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:57:51.813222 24642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:57:51.813443 24642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:57:51.813747 24642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:57:51.814107 24642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:57:51.814157 24642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:51.814217 24642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:57:51.814247 24642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:51.822340 24642 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:45175
I20260806 00:57:51.822429 24775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:45175 every 8 connection(s)
I20260806 00:57:51.822882 24642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
Aug 06 00:57:51 dist-test-slave-dhph krb5kdc[24527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977871, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:57:51.831302 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 24642
I20260806 00:57:51.831420 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance
I20260806 00:57:51.831604 24520 external_mini_cluster.cc:1428] Setting key a2c662fdc39344f2dcdf166c10af7893
I20260806 00:57:51.836431 24619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:51.824072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:49503 (local address 127.23.242.62:33117)
0806 00:57:51.824157 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:51.824160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:51.824172 (+ 12us) server_negotiation.cc:407] Connection header received
0806 00:57:51.824810 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:51.824812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:51.824826 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:51.824881 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:51.826238 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:51.826839 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:51.827858 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:51.828062 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:51.828136 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:51.831668 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:51.831671 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:51.831671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:51.831688 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:51.834009 (+ 2321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:51.834643 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:51.834645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:51.834646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:51.834724 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:51.835383 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:51.835386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:51.835386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:51.835676 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:51.835802 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:51.836245 (+ 443us) server_negotiation.cc:299] Negotiation successful
0806 00:57:51.836347 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 00:57:51.836740 24778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:51.823768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:33117 (local address 127.23.242.1:49503)
0806 00:57:51.823998 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:51.824059 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:51.824668 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:51.824956 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:51.824961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:51.825891 (+ 930us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:57:51.826104 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:51.826109 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:51.826937 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:51.826938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:51.827759 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:51.827761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:51.828160 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:51.828161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:51.828291 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:51.829363 (+ 1072us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:57:51.829374 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:57:51.831389 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:57:51.834144 (+ 2755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:51.834158 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:51.834161 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:51.834473 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:51.834812 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:51.834813 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:51.834814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:51.835207 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:51.835797 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:57:51.835809 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:57:51.836031 (+ 222us) client_negotiation.cc:769] Sending connection context
0806 00:57:51.836175 (+ 144us) client_negotiation.cc:240] Negotiation successful
0806 00:57:51.836427 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":76,"threads_started":1}
I20260806 00:57:51.837325 24776 heartbeater.cc:347] Connected to a master server at 127.23.242.62:33117
I20260806 00:57:51.837458 24776 heartbeater.cc:464] Registering TS with master...
I20260806 00:57:51.837677 24776 heartbeater.cc:511] Master 127.23.242.62:33117 requested a full tablet report, sending...
I20260806 00:57:51.838423 24574 ts_manager.cc:194] Registered new tserver with Master: 71ba6d56a343410daeb0eded35a00871 (127.23.242.1:45175)
I20260806 00:57:51.839466 24574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.1@KRBTEST.COM'} at 127.23.242.1:49503
WARNING: no policy specified for kudu/127.23.242.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
I20260806 00:57:51.872227 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:0
--local_ip_for_outbound_sockets=127.23.242.2
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:33117
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:41015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:57:51.977419 24781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:57:51.977661 24781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:57:51.977689 24781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:57:51.979122 24781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:57:51.979180 24781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:57:51.979212 24781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 00:57:51.980792 24781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:33117
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:57:51.981060 24781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:57:51.981335 24781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:57:51.987825 24786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:57:51.987818 24787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:57:51.987850 24789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:57:51.988269 24781 server_base.cc:1034] running on GCE node
I20260806 00:57:51.988600 24781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:57:51.988971 24781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:57:51.990432 24781 hybrid_clock.cc:648] HybridClock initialized: now 1785977871990310 us; error 142 us; skew 500 ppm
Aug 06 00:57:51 dist-test-slave-dhph krb5kdc[24527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977871, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:57:51.993620 24781 init.cc:377] Logged in from keytab as kudu/127.23.242.2@KRBTEST.COM (short username kudu)
I20260806 00:57:51.994117 24781 webserver.cc:460] Webserver started at http://127.23.242.2:44569/ using document root <none> and password file <none>
I20260806 00:57:51.994385 24781 fs_manager.cc:359] Metadata directory not provided
I20260806 00:57:51.994437 24781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:57:51.994639 24781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:57:51.996184 24781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-1/data/instance:
uuid: "074bacb896bb427eaa0d57b5e77e9744"
format_stamp: "Formatted at 2026-08-06 00:57:51 on dist-test-slave-dhph"
server_key: "7b9d29cca44b72a8c6f9a29a00e47a25"
server_key_iv: "83b9d13986f5fb99e5e24bcd8f83cbfc"
server_key_version: "encryptionkey@0"
I20260806 00:57:51.996613 24781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance:
uuid: "074bacb896bb427eaa0d57b5e77e9744"
format_stamp: "Formatted at 2026-08-06 00:57:51 on dist-test-slave-dhph"
server_key: "7b9d29cca44b72a8c6f9a29a00e47a25"
server_key_iv: "83b9d13986f5fb99e5e24bcd8f83cbfc"
server_key_version: "encryptionkey@0"
I20260806 00:57:51.998363 24781 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 00:57:51.999528 24797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:52.000104 24781 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 00:57:52.000197 24781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "074bacb896bb427eaa0d57b5e77e9744"
format_stamp: "Formatted at 2026-08-06 00:57:51 on dist-test-slave-dhph"
server_key: "7b9d29cca44b72a8c6f9a29a00e47a25"
server_key_iv: "83b9d13986f5fb99e5e24bcd8f83cbfc"
server_key_version: "encryptionkey@0"
I20260806 00:57:52.000260 24781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:57:52.015175 24781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:57:52.019199 24781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:57:52.019439 24781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:57:52.020056 24781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:57:52.020640 24781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:57:52.020690 24781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:52.020720 24781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:57:52.020735 24781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:52.029743 24781 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:46393
I20260806 00:57:52.029824 24912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:46393 every 8 connection(s)
I20260806 00:57:52.030232 24781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
Aug 06 00:57:52 dist-test-slave-dhph krb5kdc[24527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977871, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:57:52.040391 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 24781
I20260806 00:57:52.040547 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance
I20260806 00:57:52.040949 24520 external_mini_cluster.cc:1428] Setting key 51b703e68e615882ecd388b02ace500f
I20260806 00:57:52.044229 24619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:52.031426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:52387 (local address 127.23.242.62:33117)
0806 00:57:52.031545 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:52.031547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:52.031896 (+ 349us) server_negotiation.cc:407] Connection header received
0806 00:57:52.032651 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:52.032653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:52.032669 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:52.032722 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:52.034010 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:52.034641 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:52.036318 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:52.036565 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:52.036844 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:52.040349 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:52.040351 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:52.040352 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:52.040361 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:52.042498 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:52.043056 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:52.043057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:52.043058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:52.043131 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:52.043439 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:52.043440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:52.043440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:52.043577 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:52.043682 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:52.044017 (+ 335us) server_negotiation.cc:299] Negotiation successful
0806 00:57:52.044116 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 00:57:52.044489 24915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:52.031435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:33117 (local address 127.23.242.2:52387)
0806 00:57:52.031696 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:52.031765 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:52.032406 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:52.032831 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:52.032835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:52.033609 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:57:52.033843 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:52.033849 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:52.035130 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:52.035131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:52.036176 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:52.036180 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:52.036860 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:52.036862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:52.037029 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:52.037958 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:57:52.037968 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:57:52.040110 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:57:52.042649 (+ 2539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:52.042662 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:52.042665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:52.042961 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:52.043214 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:52.043215 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:52.043215 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:52.043351 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:52.043675 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:57:52.043682 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:57:52.043852 (+ 170us) client_negotiation.cc:769] Sending connection context
0806 00:57:52.043983 (+ 131us) client_negotiation.cc:240] Negotiation successful
0806 00:57:52.044198 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":107,"threads_started":1}
I20260806 00:57:52.045009 24913 heartbeater.cc:347] Connected to a master server at 127.23.242.62:33117
I20260806 00:57:52.045142 24913 heartbeater.cc:464] Registering TS with master...
I20260806 00:57:52.045496 24913 heartbeater.cc:511] Master 127.23.242.62:33117 requested a full tablet report, sending...
I20260806 00:57:52.046766 24574 ts_manager.cc:194] Registered new tserver with Master: 074bacb896bb427eaa0d57b5e77e9744 (127.23.242.2:46393)
I20260806 00:57:52.047210 24574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.2@KRBTEST.COM'} at 127.23.242.2:52387
WARNING: no policy specified for kudu/127.23.242.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
I20260806 00:57:52.082927 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:0
--local_ip_for_outbound_sockets=127.23.242.3
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:33117
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:41015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:57:52.191910 24918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:57:52.192169 24918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:57:52.192202 24918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:57:52.193709 24918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:57:52.193770 24918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:57:52.193805 24918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 00:57:52.196127 24918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:33117
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:57:52.196455 24918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:57:52.196767 24918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:57:52.203358 24927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:57:52.203460 24923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:57:52.203358 24925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:57:52.203729 24918 server_base.cc:1034] running on GCE node
I20260806 00:57:52.204134 24918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:57:52.204862 24918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:57:52.206116 24918 hybrid_clock.cc:648] HybridClock initialized: now 1785977872206102 us; error 37 us; skew 500 ppm
Aug 06 00:57:52 dist-test-slave-dhph krb5kdc[24527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977872, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:57:52.209460 24918 init.cc:377] Logged in from keytab as kudu/127.23.242.3@KRBTEST.COM (short username kudu)
I20260806 00:57:52.210001 24918 webserver.cc:460] Webserver started at http://127.23.242.3:46319/ using document root <none> and password file <none>
I20260806 00:57:52.210418 24918 fs_manager.cc:359] Metadata directory not provided
I20260806 00:57:52.210523 24918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:57:52.210731 24918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:57:52.212177 24918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-2/data/instance:
uuid: "353d316f06f349fd8e1b64146d5c1dc4"
format_stamp: "Formatted at 2026-08-06 00:57:52 on dist-test-slave-dhph"
server_key: "da0e4ef15baa32efb8ffaf342601084e"
server_key_iv: "b0ee128f519c6af775c1f42e83e6e33e"
server_key_version: "encryptionkey@0"
I20260806 00:57:52.213006 24918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance:
uuid: "353d316f06f349fd8e1b64146d5c1dc4"
format_stamp: "Formatted at 2026-08-06 00:57:52 on dist-test-slave-dhph"
server_key: "da0e4ef15baa32efb8ffaf342601084e"
server_key_iv: "b0ee128f519c6af775c1f42e83e6e33e"
server_key_version: "encryptionkey@0"
I20260806 00:57:52.216617 24918 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260806 00:57:52.218374 24934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:52.219345 24918 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 00:57:52.219738 24918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "353d316f06f349fd8e1b64146d5c1dc4"
format_stamp: "Formatted at 2026-08-06 00:57:52 on dist-test-slave-dhph"
server_key: "da0e4ef15baa32efb8ffaf342601084e"
server_key_iv: "b0ee128f519c6af775c1f42e83e6e33e"
server_key_version: "encryptionkey@0"
I20260806 00:57:52.219868 24918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:57:52.237493 24918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:57:52.241272 24918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:57:52.241842 24918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:57:52.242368 24918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:57:52.242796 24918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:57:52.242853 24918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:52.242888 24918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:57:52.242934 24918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:52.251415 24918 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:36635
I20260806 00:57:52.251471 25048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:36635 every 8 connection(s)
I20260806 00:57:52.252017 24918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
I20260806 00:57:52.261716 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 24918
I20260806 00:57:52.261816 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance
I20260806 00:57:52.261961 24520 external_mini_cluster.cc:1428] Setting key f02464db718018c592d5851e0c2b2264
Aug 06 00:57:52 dist-test-slave-dhph krb5kdc[24527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977872, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:57:52.269338 24619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:52.252950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:48545 (local address 127.23.242.62:33117)
0806 00:57:52.253005 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:52.253007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:52.253243 (+ 236us) server_negotiation.cc:407] Connection header received
0806 00:57:52.253761 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:52.253762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:52.253774 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:52.253849 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:52.255083 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:52.255784 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:52.256925 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:52.257194 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:52.257270 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:52.265860 (+ 8590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:52.265861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:52.265861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:52.265870 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:52.267624 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:52.268134 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:52.268135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:52.268135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:52.268209 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:52.268555 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:52.268556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:52.268557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:52.268747 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:52.268815 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:52.269179 (+ 364us) server_negotiation.cc:299] Negotiation successful
0806 00:57:52.269260 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 00:57:52.269469 25051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:52.252862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:33117 (local address 127.23.242.3:48545)
0806 00:57:52.253040 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:52.253086 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:52.253618 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:52.253959 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:52.253963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:52.254703 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:57:52.254927 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:52.254933 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:52.255929 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:52.255930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:52.256785 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:52.256788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:52.257348 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:52.257349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:52.257466 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:52.263418 (+ 5952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:57:52.263439 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:57:52.265654 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:57:52.267739 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:52.267748 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:52.267750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:52.268024 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:52.268319 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:52.268320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:52.268320 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:52.268460 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:52.268827 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:57:52.268834 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:57:52.268979 (+ 145us) client_negotiation.cc:769] Sending connection context
0806 00:57:52.269086 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 00:57:52.269272 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
I20260806 00:57:52.270237 25049 heartbeater.cc:347] Connected to a master server at 127.23.242.62:33117
I20260806 00:57:52.270344 25049 heartbeater.cc:464] Registering TS with master...
I20260806 00:57:52.270500 25049 heartbeater.cc:511] Master 127.23.242.62:33117 requested a full tablet report, sending...
I20260806 00:57:52.270993 24574 ts_manager.cc:194] Registered new tserver with Master: 353d316f06f349fd8e1b64146d5c1dc4 (127.23.242.3:36635)
I20260806 00:57:52.271389 24574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.3@KRBTEST.COM'} at 127.23.242.3:48545
I20260806 00:57:52.275954 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 00:57:52 dist-test-slave-dhph krb5kdc[24527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977872, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 00:57:52 dist-test-slave-dhph krb5kdc[24527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977872, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.242.1@KRBTEST.COM
I20260806 00:57:52.307917 25056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:52.296163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42518 (local address 127.23.242.1:45175)
0806 00:57:52.296327 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:52.296332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:52.296345 (+ 13us) server_negotiation.cc:407] Connection header received
0806 00:57:52.296417 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:52.296419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:52.296436 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:52.296514 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:52.297491 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:52.298400 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:52.299684 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:52.300153 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:52.300251 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:52.303703 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:52.303720 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:52.303720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:52.303731 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:52.306095 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:52.306746 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:52.306749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:52.306749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:52.306834 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:52.307204 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:52.307205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:52.307205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:52.307477 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:52.307622 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:52.307727 (+ 105us) server_negotiation.cc:299] Negotiation successful
0806 00:57:52.307818 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
W20260806 00:57:52.308629 24688 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42518
Aug 06 00:57:52 dist-test-slave-dhph krb5kdc[24527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 00:57:52 dist-test-slave-dhph krb5kdc[24527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.1@KRBTEST.COM
I20260806 00:57:52.340401 25056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:52.329631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42528 (local address 127.23.242.1:45175)
0806 00:57:52.329745 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:52.329747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:52.329760 (+ 13us) server_negotiation.cc:407] Connection header received
0806 00:57:52.329796 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:52.329797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:52.329813 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:52.329912 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:52.330784 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:52.331365 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:52.332334 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:52.332597 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:52.332669 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:52.335964 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:52.335967 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:52.335968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:52.335980 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:52.337994 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:52.338971 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:52.338972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:52.338973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:52.339045 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:52.339339 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:52.339340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:52.339340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:52.339462 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:52.339551 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:52.340232 (+ 681us) server_negotiation.cc:299] Negotiation successful
0806 00:57:52.340327 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 00:57:52.341315 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 24642
I20260806 00:57:52.349660 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 24781
I20260806 00:57:52.355445 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 24918
I20260806 00:57:52.363312 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 24542
2026-08-06T00:57:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2382 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 00:57:52 dist-test-slave-dhph krb5kdc[25063](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 00:57:52 dist-test-slave-dhph krb5kdc[25063](info): set up 2 sockets
Aug 06 00:57:52 dist-test-slave-dhph krb5kdc[25063](info): commencing operation
krb5kdc: starting...
W20260806 00:57:53.586272 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 00:57:53 dist-test-slave-dhph krb5kdc[25063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T00:57:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T00:57:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 00:57:53.741729 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:44739
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:44739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:57:53.834211 25076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:57:53.834619 25076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:57:53.834671 25076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:57:53.835880 25076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 00:57:53.835963 25076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:57:53.836009 25076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:57:53.836045 25076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 00:57:53.836076 25076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 00:57:53.837695 25076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:44739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:44739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:57:53.838181 25076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:57:53.838488 25076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:57:53.844538 25083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:57:53.844595 25076 server_base.cc:1034] running on GCE node
W20260806 00:57:53.844614 25084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:57:53.844538 25086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:57:53.845245 25076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:57:53.845567 25076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:57:53.846738 25076 hybrid_clock.cc:648] HybridClock initialized: now 1785977873846713 us; error 51 us; skew 500 ppm
Aug 06 00:57:53 dist-test-slave-dhph krb5kdc[25063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977873, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:57:53.850306 25076 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 00:57:53.850992 25076 webserver.cc:460] Webserver started at http://127.23.242.62:42755/ using document root <none> and password file <none>
I20260806 00:57:53.851301 25076 fs_manager.cc:359] Metadata directory not provided
I20260806 00:57:53.851384 25076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:57:53.851557 25076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:57:53.852799 25076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "5b9be179b344455aac93b0d751254180"
format_stamp: "Formatted at 2026-08-06 00:57:53 on dist-test-slave-dhph"
server_key: "b6b96ad0ffec3ab950bc5faf8384dd6e"
server_key_iv: "bd49d436ab8fcc481218d38c2e042db8"
server_key_version: "encryptionkey@0"
I20260806 00:57:53.853361 25076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "5b9be179b344455aac93b0d751254180"
format_stamp: "Formatted at 2026-08-06 00:57:53 on dist-test-slave-dhph"
server_key: "b6b96ad0ffec3ab950bc5faf8384dd6e"
server_key_iv: "bd49d436ab8fcc481218d38c2e042db8"
server_key_version: "encryptionkey@0"
I20260806 00:57:53.855719 25076 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260806 00:57:53.856895 25093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:53.857352 25076 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:57:53.857491 25076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "5b9be179b344455aac93b0d751254180"
format_stamp: "Formatted at 2026-08-06 00:57:53 on dist-test-slave-dhph"
server_key: "b6b96ad0ffec3ab950bc5faf8384dd6e"
server_key_iv: "bd49d436ab8fcc481218d38c2e042db8"
server_key_version: "encryptionkey@0"
I20260806 00:57:53.857606 25076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:57:53.870136 25076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:57:53.872985 25076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:57:53.873184 25076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:57:53.877966 25076 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:44739
I20260806 00:57:53.878010 25148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:44739 every 8 connection(s)
I20260806 00:57:53.878407 25076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 00:57:53.878732 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 25076
I20260806 00:57:53.878818 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 00:57:53.879324 24520 external_mini_cluster.cc:1428] Setting key 9c9340fad5c610937a967585a9aef744
I20260806 00:57:53.880213 25149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:57:53.883251 25149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b9be179b344455aac93b0d751254180: Bootstrap starting.
I20260806 00:57:53.883981 25149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b9be179b344455aac93b0d751254180: Neither blocks nor log segments found. Creating new log.
I20260806 00:57:53.884336 25149 log.cc:824] T 00000000000000000000000000000000 P 5b9be179b344455aac93b0d751254180: Log is configured to *not* fsync() on all Append() calls
I20260806 00:57:53.885273 25149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b9be179b344455aac93b0d751254180: No bootstrap required, opened a new log
Aug 06 00:57:53 dist-test-slave-dhph krb5kdc[25063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:57:53.888054 25149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b9be179b344455aac93b0d751254180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9be179b344455aac93b0d751254180" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44739 } }
I20260806 00:57:53.888265 25149 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b9be179b344455aac93b0d751254180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:57:53.888319 25149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b9be179b344455aac93b0d751254180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b9be179b344455aac93b0d751254180, State: Initialized, Role: FOLLOWER
I20260806 00:57:53.888484 25149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b9be179b344455aac93b0d751254180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9be179b344455aac93b0d751254180" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44739 } }
I20260806 00:57:53.888592 25149 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b9be179b344455aac93b0d751254180 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 00:57:53.888644 25149 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b9be179b344455aac93b0d751254180 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 00:57:53.888700 25149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b9be179b344455aac93b0d751254180 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:57:53.889614 25149 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b9be179b344455aac93b0d751254180 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9be179b344455aac93b0d751254180" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44739 } }
I20260806 00:57:53.889751 25149 leader_election.cc:302] T 00000000000000000000000000000000 P 5b9be179b344455aac93b0d751254180 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b9be179b344455aac93b0d751254180; no voters:
I20260806 00:57:53.889982 25149 leader_election.cc:288] T 00000000000000000000000000000000 P 5b9be179b344455aac93b0d751254180 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 00:57:53.890184 25154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b9be179b344455aac93b0d751254180 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:57:53.890316 25149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b9be179b344455aac93b0d751254180 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 00:57:53.890552 25154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b9be179b344455aac93b0d751254180 [term 1 LEADER]: Becoming Leader. State: Replica: 5b9be179b344455aac93b0d751254180, State: Running, Role: LEADER
I20260806 00:57:53.890784 25154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b9be179b344455aac93b0d751254180 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9be179b344455aac93b0d751254180" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44739 } }
I20260806 00:57:53.891323 25154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b9be179b344455aac93b0d751254180 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b9be179b344455aac93b0d751254180. Latest consensus state: current_term: 1 leader_uuid: "5b9be179b344455aac93b0d751254180" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9be179b344455aac93b0d751254180" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44739 } } }
I20260806 00:57:53.891450 25154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b9be179b344455aac93b0d751254180 [sys.catalog]: This master's current role is: LEADER
I20260806 00:57:53.891328 25156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b9be179b344455aac93b0d751254180 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b9be179b344455aac93b0d751254180" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9be179b344455aac93b0d751254180" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44739 } } }
I20260806 00:57:53.891691 25156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b9be179b344455aac93b0d751254180 [sys.catalog]: This master's current role is: LEADER
I20260806 00:57:53.891847 25165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 00:57:53.892693 25165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 00:57:53.894475 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:53.879989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59986 (local address 127.23.242.62:44739)
0806 00:57:53.880224 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:53.880242 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:53.880297 (+ 55us) server_negotiation.cc:407] Connection header received
0806 00:57:53.880878 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:53.880887 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:53.880940 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:53.881304 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:53.882076 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:53.883115 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:53.884029 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:53.884621 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:53.884764 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:53.887960 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:53.887964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:53.887964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:53.887977 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:53.891740 (+ 3763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:53.892406 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:53.892411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:53.892412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:53.892573 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:53.892898 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:53.892900 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:53.892900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:53.893367 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:53.893977 (+ 610us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:53.894075 (+ 98us) server_negotiation.cc:299] Negotiation successful
0806 00:57:53.894246 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":115,"threads_started":1}
I20260806 00:57:53.895159 25165 catalog_manager.cc:1347] Generated new cluster ID: 7c38803fdd5b413f81a3b2522594f3c8
I20260806 00:57:53.895226 25165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 00:57:53.919137 25165 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 00:57:53.919783 25165 catalog_manager.cc:1504] Loading token signing keys...
I20260806 00:57:53.926254 25165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b9be179b344455aac93b0d751254180: Generated new TSK 0
I20260806 00:57:53.926688 25165 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.242.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
I20260806 00:57:53.968770 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:0
--local_ip_for_outbound_sockets=127.23.242.1
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:44739
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:57:54.073457 25176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:57:54.073899 25176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:57:54.073939 25176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:57:54.075904 25176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:57:54.075964 25176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:57:54.075999 25176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 00:57:54.077796 25176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:44739
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:57:54.078017 25176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:57:54.078227 25176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 00:57:54.084995 25176 server_base.cc:1034] running on GCE node
W20260806 00:57:54.084954 25182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:57:54.084955 25184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:57:54.084955 25181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:57:54.085520 25176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:57:54.085950 25176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:57:54.087173 25176 hybrid_clock.cc:648] HybridClock initialized: now 1785977874087107 us; error 89 us; skew 500 ppm
Aug 06 00:57:54 dist-test-slave-dhph krb5kdc[25063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977874, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:57:54.090162 25176 init.cc:377] Logged in from keytab as kudu/127.23.242.1@KRBTEST.COM (short username kudu)
I20260806 00:57:54.090816 25176 webserver.cc:460] Webserver started at http://127.23.242.1:45815/ using document root <none> and password file <none>
I20260806 00:57:54.091065 25176 fs_manager.cc:359] Metadata directory not provided
I20260806 00:57:54.091114 25176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:57:54.091243 25176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:57:54.092499 25176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-0/data/instance:
uuid: "120c99c8c0224870a82735cd7ceba2b7"
format_stamp: "Formatted at 2026-08-06 00:57:54 on dist-test-slave-dhph"
server_key: "f7a9bc22fc703d50f9bd9c8de7137f27"
server_key_iv: "98cf6073287fc95b578ff2349a93c138"
server_key_version: "encryptionkey@0"
I20260806 00:57:54.093727 25176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance:
uuid: "120c99c8c0224870a82735cd7ceba2b7"
format_stamp: "Formatted at 2026-08-06 00:57:54 on dist-test-slave-dhph"
server_key: "f7a9bc22fc703d50f9bd9c8de7137f27"
server_key_iv: "98cf6073287fc95b578ff2349a93c138"
server_key_version: "encryptionkey@0"
I20260806 00:57:54.096122 25176 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.004s
I20260806 00:57:54.097288 25191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:54.097759 25176 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:57:54.097856 25176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "120c99c8c0224870a82735cd7ceba2b7"
format_stamp: "Formatted at 2026-08-06 00:57:54 on dist-test-slave-dhph"
server_key: "f7a9bc22fc703d50f9bd9c8de7137f27"
server_key_iv: "98cf6073287fc95b578ff2349a93c138"
server_key_version: "encryptionkey@0"
I20260806 00:57:54.097970 25176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:57:54.109655 25176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:57:54.112831 25176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:57:54.113529 25176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:57:54.114096 25176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:57:54.114521 25176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:57:54.114563 25176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:54.114657 25176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:57:54.114696 25176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:54.123448 25176 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:42337
I20260806 00:57:54.123548 25306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:42337 every 8 connection(s)
I20260806 00:57:54.123896 25176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 00:57:54.126294 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 25176
I20260806 00:57:54.126417 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance
I20260806 00:57:54.126688 24520 external_mini_cluster.cc:1428] Setting key dd839608d65a177ad397b6a7cd39550d
Aug 06 00:57:54 dist-test-slave-dhph krb5kdc[25063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977874, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.242.2@KRBTEST.COM; defaulting to no policy
I20260806 00:57:54.139451 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:54.124889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:39555 (local address 127.23.242.62:44739)
0806 00:57:54.124961 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:54.124963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:54.125218 (+ 255us) server_negotiation.cc:407] Connection header received
0806 00:57:54.127087 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:54.127089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:54.127101 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:54.127202 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:54.128111 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.128869 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.131754 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.132008 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.132093 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:54.135441 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:54.135444 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:54.135446 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:54.135460 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:54.137058 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:54.137788 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:54.137790 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:54.137791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:54.137976 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:54.138539 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:54.138540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:54.138541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:54.138790 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:54.138888 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:54.139218 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 00:57:54.139334 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 00:57:54.139665 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:54.124829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:44739 (local address 127.23.242.1:39555)
0806 00:57:54.125016 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:54.125065 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:54.126920 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:54.127309 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:54.127315 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:54.127761 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:57:54.127994 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:54.127999 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.129052 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.129055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:54.131214 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:54.131221 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.132187 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.132189 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:54.132379 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:54.133064 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:57:54.133077 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:57:54.135317 (+ 2240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:57:54.137223 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:54.137232 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:54.137236 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:54.137501 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:54.138183 (+ 682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:54.138185 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:54.138185 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:54.138429 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:54.138876 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:57:54.138885 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:57:54.139058 (+ 173us) client_negotiation.cc:769] Sending connection context
0806 00:57:54.139180 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 00:57:54.139394 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":104,"threads_started":1}
I20260806 00:57:54.140554 25307 heartbeater.cc:347] Connected to a master server at 127.23.242.62:44739
I20260806 00:57:54.140704 25307 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.2@KRBTEST.COM" created.
I20260806 00:57:54.140939 25307 heartbeater.cc:511] Master 127.23.242.62:44739 requested a full tablet report, sending...
I20260806 00:57:54.141698 25111 ts_manager.cc:194] Registered new tserver with Master: 120c99c8c0224870a82735cd7ceba2b7 (127.23.242.1:42337)
I20260806 00:57:54.142463 25111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.1@KRBTEST.COM'} at 127.23.242.1:39555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
I20260806 00:57:54.172120 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:0
--local_ip_for_outbound_sockets=127.23.242.2
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:44739
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:57:54.272372 25312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:57:54.272624 25312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:57:54.272665 25312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:57:54.273844 25312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:57:54.273921 25312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:57:54.273968 25312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 00:57:54.275591 25312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:44739
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:57:54.275882 25312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:57:54.276120 25312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:57:54.281549 25319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:57:54.281559 25318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:57:54.281550 25321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:57:54.282089 25312 server_base.cc:1034] running on GCE node
I20260806 00:57:54.282366 25312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:57:54.282701 25312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:57:54.283864 25312 hybrid_clock.cc:648] HybridClock initialized: now 1785977874283839 us; error 43 us; skew 500 ppm
Aug 06 00:57:54 dist-test-slave-dhph krb5kdc[25063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977874, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:57:54.287335 25312 init.cc:377] Logged in from keytab as kudu/127.23.242.2@KRBTEST.COM (short username kudu)
I20260806 00:57:54.287902 25312 webserver.cc:460] Webserver started at http://127.23.242.2:33159/ using document root <none> and password file <none>
I20260806 00:57:54.288182 25312 fs_manager.cc:359] Metadata directory not provided
I20260806 00:57:54.288268 25312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:57:54.288432 25312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:57:54.289670 25312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-1/data/instance:
uuid: "700a76bf74bd4c54bf201448dee49658"
format_stamp: "Formatted at 2026-08-06 00:57:54 on dist-test-slave-dhph"
server_key: "2197b0c6981ab39b2e5580c9ffc9111c"
server_key_iv: "d5e7d2f722b0216217ee9571320c50f2"
server_key_version: "encryptionkey@0"
I20260806 00:57:54.290275 25312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance:
uuid: "700a76bf74bd4c54bf201448dee49658"
format_stamp: "Formatted at 2026-08-06 00:57:54 on dist-test-slave-dhph"
server_key: "2197b0c6981ab39b2e5580c9ffc9111c"
server_key_iv: "d5e7d2f722b0216217ee9571320c50f2"
server_key_version: "encryptionkey@0"
I20260806 00:57:54.292994 25312 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 00:57:54.293941 25328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:54.294229 25312 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:57:54.294314 25312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "700a76bf74bd4c54bf201448dee49658"
format_stamp: "Formatted at 2026-08-06 00:57:54 on dist-test-slave-dhph"
server_key: "2197b0c6981ab39b2e5580c9ffc9111c"
server_key_iv: "d5e7d2f722b0216217ee9571320c50f2"
server_key_version: "encryptionkey@0"
I20260806 00:57:54.294377 25312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:57:54.310990 25312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:57:54.313846 25312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:57:54.313984 25312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:57:54.314327 25312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:57:54.314720 25312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:57:54.314759 25312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:54.314785 25312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:57:54.314801 25312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:54.322085 25312 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:37267
I20260806 00:57:54.322196 25442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:37267 every 8 connection(s)
I20260806 00:57:54.322508 25312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 00:57:54.329543 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 25312
I20260806 00:57:54.329658 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance
I20260806 00:57:54.329887 24520 external_mini_cluster.cc:1428] Setting key 0bbd9aecb23099b1047faae3d5e33b36
Aug 06 00:57:54 dist-test-slave-dhph krb5kdc[25063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977874, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:57:54.336288 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:54.323505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:53613 (local address 127.23.242.62:44739)
0806 00:57:54.323785 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:54.323787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:54.323993 (+ 206us) server_negotiation.cc:407] Connection header received
0806 00:57:54.324963 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:54.324965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:54.324977 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:54.325054 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:54.325931 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.326447 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.327702 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.327908 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.327979 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:54.331424 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:54.331426 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:54.331426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:54.331464 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:54.334273 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:54.334952 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:54.334954 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:54.334954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:54.335022 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:54.335384 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:54.335385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:54.335385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:54.335529 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:54.335589 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:54.336060 (+ 471us) server_negotiation.cc:299] Negotiation successful
0806 00:57:54.336149 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238}
I20260806 00:57:54.336481 25445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:54.323413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:44739 (local address 127.23.242.2:53613)
0806 00:57:54.323741 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:54.323803 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:54.324831 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:54.325216 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:54.325220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:54.325524 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:57:54.325746 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:54.325750 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.326597 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.326599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:54.327562 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:54.327564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.328212 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.328214 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:54.328470 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:54.329303 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:57:54.329317 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:57:54.331215 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:57:54.334382 (+ 3167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:54.334393 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:54.334396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:54.334842 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:54.335091 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:54.335092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:54.335093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:54.335265 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:54.335610 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:57:54.335618 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:57:54.335816 (+ 198us) client_negotiation.cc:769] Sending connection context
0806 00:57:54.335942 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 00:57:54.336173 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":82,"threads_started":1}
I20260806 00:57:54.337914 25443 heartbeater.cc:347] Connected to a master server at 127.23.242.62:44739
I20260806 00:57:54.338040 25443 heartbeater.cc:464] Registering TS with master...
I20260806 00:57:54.338265 25443 heartbeater.cc:511] Master 127.23.242.62:44739 requested a full tablet report, sending...
I20260806 00:57:54.338749 25111 ts_manager.cc:194] Registered new tserver with Master: 700a76bf74bd4c54bf201448dee49658 (127.23.242.2:37267)
I20260806 00:57:54.339151 25111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.2@KRBTEST.COM'} at 127.23.242.2:53613
WARNING: no policy specified for kudu/127.23.242.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
I20260806 00:57:54.371311 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:0
--local_ip_for_outbound_sockets=127.23.242.3
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:44739
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:41803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:57:54.467967 25448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:57:54.468848 25448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:57:54.468914 25448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:57:54.470418 25448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:57:54.470468 25448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:57:54.470498 25448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 00:57:54.472162 25448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:44739
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:57:54.472424 25448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:57:54.472671 25448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:57:54.478103 25456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:57:54.478103 25454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:57:54.478299 25448 server_base.cc:1034] running on GCE node
W20260806 00:57:54.478109 25453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:57:54.478649 25448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:57:54.478951 25448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:57:54.480147 25448 hybrid_clock.cc:648] HybridClock initialized: now 1785977874480107 us; error 31 us; skew 500 ppm
Aug 06 00:57:54 dist-test-slave-dhph krb5kdc[25063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977874, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:57:54.483180 25448 init.cc:377] Logged in from keytab as kudu/127.23.242.3@KRBTEST.COM (short username kudu)
I20260806 00:57:54.483702 25448 webserver.cc:460] Webserver started at http://127.23.242.3:36651/ using document root <none> and password file <none>
I20260806 00:57:54.483943 25448 fs_manager.cc:359] Metadata directory not provided
I20260806 00:57:54.484011 25448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:57:54.484186 25448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:57:54.485626 25448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-2/data/instance:
uuid: "d884f2f770024d05b6c6108a691a8e23"
format_stamp: "Formatted at 2026-08-06 00:57:54 on dist-test-slave-dhph"
server_key: "cd2385aeea21ba853c8dbb74e4082f6e"
server_key_iv: "dbba42ee22874c9e1a0744e54ada4862"
server_key_version: "encryptionkey@0"
I20260806 00:57:54.486018 25448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance:
uuid: "d884f2f770024d05b6c6108a691a8e23"
format_stamp: "Formatted at 2026-08-06 00:57:54 on dist-test-slave-dhph"
server_key: "cd2385aeea21ba853c8dbb74e4082f6e"
server_key_iv: "dbba42ee22874c9e1a0744e54ada4862"
server_key_version: "encryptionkey@0"
I20260806 00:57:54.487583 25448 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260806 00:57:54.488469 25463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:54.488682 25448 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 00:57:54.488837 25448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "d884f2f770024d05b6c6108a691a8e23"
format_stamp: "Formatted at 2026-08-06 00:57:54 on dist-test-slave-dhph"
server_key: "cd2385aeea21ba853c8dbb74e4082f6e"
server_key_iv: "dbba42ee22874c9e1a0744e54ada4862"
server_key_version: "encryptionkey@0"
I20260806 00:57:54.488919 25448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:57:54.517714 25448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:57:54.520583 25448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:57:54.520789 25448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:57:54.521091 25448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:57:54.521445 25448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:57:54.521505 25448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:54.521552 25448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:57:54.521600 25448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:54.530068 25448 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:32863
I20260806 00:57:54.530196 25579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:32863 every 8 connection(s)
I20260806 00:57:54.530565 25448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
Aug 06 00:57:54 dist-test-slave-dhph krb5kdc[25063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977874, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:57:54.538929 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 25448
I20260806 00:57:54.539036 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance
I20260806 00:57:54.539216 24520 external_mini_cluster.cc:1428] Setting key e709af84c00b90af16a7915ece220544
I20260806 00:57:54.543460 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:54.531590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:47857 (local address 127.23.242.62:44739)
0806 00:57:54.531649 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:54.531652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:54.531780 (+ 128us) server_negotiation.cc:407] Connection header received
0806 00:57:54.532870 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:54.532871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:54.532885 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:54.532957 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:54.533715 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.534254 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.535800 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.536291 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.536375 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:54.539342 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:54.539345 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:54.539346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:54.539356 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:54.541414 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:54.542102 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:54.542104 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:54.542104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:54.542168 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:54.542572 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:54.542573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:54.542573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:54.542785 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:54.542850 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:54.543277 (+ 427us) server_negotiation.cc:299] Negotiation successful
0806 00:57:54.543382 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 00:57:54.543736 25582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:54.531460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:44739 (local address 127.23.242.3:47857)
0806 00:57:54.531652 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:54.531699 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:54.532708 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:54.533093 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:54.533097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:54.533397 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:57:54.533596 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:54.533600 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.534364 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.534365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:54.535508 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:54.535511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.536438 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.536439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:54.536560 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:54.537407 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:57:54.537416 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:57:54.539178 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:57:54.541677 (+ 2499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:54.541691 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:54.541694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:54.541996 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:54.542336 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:54.542337 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:54.542338 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:54.542466 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:54.542923 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:57:54.542931 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:57:54.543076 (+ 145us) client_negotiation.cc:769] Sending connection context
0806 00:57:54.543208 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 00:57:54.543384 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":110,"threads_started":1}
I20260806 00:57:54.544625 25580 heartbeater.cc:347] Connected to a master server at 127.23.242.62:44739
I20260806 00:57:54.544744 25580 heartbeater.cc:464] Registering TS with master...
I20260806 00:57:54.544975 25580 heartbeater.cc:511] Master 127.23.242.62:44739 requested a full tablet report, sending...
I20260806 00:57:54.545547 25111 ts_manager.cc:194] Registered new tserver with Master: d884f2f770024d05b6c6108a691a8e23 (127.23.242.3:32863)
I20260806 00:57:54.546082 25111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.3@KRBTEST.COM'} at 127.23.242.3:47857
I20260806 00:57:54.552809 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 00:57:54.563880 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:54.554752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60002 (local address 127.23.242.62:44739)
0806 00:57:54.554826 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:54.554828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:54.555399 (+ 571us) server_negotiation.cc:407] Connection header received
0806 00:57:54.555507 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:54.555508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:54.555520 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:54.555575 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:54.556235 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.556918 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.558309 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.558706 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.558911 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:54.560792 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:54.560793 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:54.560793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:54.560801 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:54.562343 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:54.562972 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:54.562973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:54.562974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:54.563036 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:54.563304 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:54.563305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:54.563305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:54.563481 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:54.563574 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:54.563727 (+ 153us) server_negotiation.cc:299] Negotiation successful
0806 00:57:54.563796 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 00:57:54.566846 25111 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60002:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 00:57:54.568616 25111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 00:57:54.577020 25593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:54.572194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:32863 (local address 127.0.0.1:40170)
0806 00:57:54.572554 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:54.572600 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:54.572678 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:54.573203 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:54.573207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:54.573210 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:57:54.573320 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:54.573321 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.574552 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.574553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:54.575754 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:54.575757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.576667 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.576668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:54.576755 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:54.576771 (+ 16us) client_negotiation.cc:769] Sending connection context
0806 00:57:54.576855 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 00:57:54.576924 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":32,"thread_start_us":79,"threads_started":1}
I20260806 00:57:54.577028 25592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:54.572299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40170 (local address 127.23.242.3:32863)
0806 00:57:54.572921 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:54.572926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:54.572936 (+ 10us) server_negotiation.cc:407] Connection header received
0806 00:57:54.572981 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:54.572984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:54.573003 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:54.573086 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:57:54.573439 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.574392 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.575859 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.576567 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.576648 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:54.576712 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:54.576886 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 00:57:54.576934 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":95,"threads_started":1}
I20260806 00:57:54.577170 25594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:54.572327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:37267 (local address 127.0.0.1:52882)
0806 00:57:54.572554 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:54.572598 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:54.572671 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:54.573206 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:54.573209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:54.573211 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:57:54.573301 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:54.573303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.574841 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.574841 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:54.576152 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:54.576155 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.576954 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.576955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:54.577020 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:54.577024 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 00:57:54.577054 (+ 30us) client_negotiation.cc:240] Negotiation successful
0806 00:57:54.577095 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":49,"thread_start_us":97,"threads_started":1}
I20260806 00:57:54.577497 25595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:54.572460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52882 (local address 127.23.242.2:37267)
0806 00:57:54.572833 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:54.572838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:54.572850 (+ 12us) server_negotiation.cc:407] Connection header received
0806 00:57:54.572981 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:54.572984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:54.573096 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:54.573407 (+ 311us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:57:54.573558 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.574698 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.576260 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.576865 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.577271 (+ 406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:54.577335 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:54.577369 (+ 34us) server_negotiation.cc:299] Negotiation successful
0806 00:57:54.577400 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":96,"threads_started":1}
I20260806 00:57:54.577613 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:54.572666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51452 (local address 127.23.242.1:42337)
0806 00:57:54.572873 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:54.572878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:54.572979 (+ 101us) server_negotiation.cc:407] Connection header received
0806 00:57:54.573108 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:54.573110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:54.573127 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:54.573204 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:57:54.573407 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.574729 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.576258 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.577098 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.577180 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:54.577260 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:54.577467 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 00:57:54.577518 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":102,"threads_started":1}
I20260806 00:57:54.577615 25591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:54.572425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:42337 (local address 127.0.0.1:51452)
0806 00:57:54.572917 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:54.572926 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:54.573005 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:54.573219 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:54.573220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:54.573222 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:57:54.573319 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:54.573321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.574839 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.574840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:54.576152 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:54.576155 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.577306 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.577307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:54.577372 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:54.577375 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 00:57:54.577433 (+ 58us) client_negotiation.cc:240] Negotiation successful
0806 00:57:54.577476 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":129,"thread_start_us":75,"threads_started":1}
I20260806 00:57:54.578392 25238 tablet_service.cc:1467] Processing CreateTablet for tablet 3fcb8e8b470b4f83ad84609f60d02573 (DEFAULT_TABLE table=test-table [id=f912d074dbbd4202bc30a40255f4d5b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:57:54.578495 25377 tablet_service.cc:1467] Processing CreateTablet for tablet 3fcb8e8b470b4f83ad84609f60d02573 (DEFAULT_TABLE table=test-table [id=f912d074dbbd4202bc30a40255f4d5b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:57:54.578864 25238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fcb8e8b470b4f83ad84609f60d02573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:57:54.578896 25377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fcb8e8b470b4f83ad84609f60d02573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:57:54.578851 25511 tablet_service.cc:1467] Processing CreateTablet for tablet 3fcb8e8b470b4f83ad84609f60d02573 (DEFAULT_TABLE table=test-table [id=f912d074dbbd4202bc30a40255f4d5b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:57:54.579169 25511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fcb8e8b470b4f83ad84609f60d02573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:57:54.582012 25597 tablet_bootstrap.cc:492] T 3fcb8e8b470b4f83ad84609f60d02573 P 700a76bf74bd4c54bf201448dee49658: Bootstrap starting.
I20260806 00:57:54.582388 25598 tablet_bootstrap.cc:492] T 3fcb8e8b470b4f83ad84609f60d02573 P 120c99c8c0224870a82735cd7ceba2b7: Bootstrap starting.
I20260806 00:57:54.582741 25599 tablet_bootstrap.cc:492] T 3fcb8e8b470b4f83ad84609f60d02573 P d884f2f770024d05b6c6108a691a8e23: Bootstrap starting.
I20260806 00:57:54.583117 25598 tablet_bootstrap.cc:654] T 3fcb8e8b470b4f83ad84609f60d02573 P 120c99c8c0224870a82735cd7ceba2b7: Neither blocks nor log segments found. Creating new log.
I20260806 00:57:54.583263 25597 tablet_bootstrap.cc:654] T 3fcb8e8b470b4f83ad84609f60d02573 P 700a76bf74bd4c54bf201448dee49658: Neither blocks nor log segments found. Creating new log.
I20260806 00:57:54.583385 25599 tablet_bootstrap.cc:654] T 3fcb8e8b470b4f83ad84609f60d02573 P d884f2f770024d05b6c6108a691a8e23: Neither blocks nor log segments found. Creating new log.
I20260806 00:57:54.583837 25598 log.cc:824] T 3fcb8e8b470b4f83ad84609f60d02573 P 120c99c8c0224870a82735cd7ceba2b7: Log is configured to *not* fsync() on all Append() calls
I20260806 00:57:54.583876 25597 log.cc:824] T 3fcb8e8b470b4f83ad84609f60d02573 P 700a76bf74bd4c54bf201448dee49658: Log is configured to *not* fsync() on all Append() calls
I20260806 00:57:54.583981 25599 log.cc:824] T 3fcb8e8b470b4f83ad84609f60d02573 P d884f2f770024d05b6c6108a691a8e23: Log is configured to *not* fsync() on all Append() calls
I20260806 00:57:54.584762 25597 tablet_bootstrap.cc:492] T 3fcb8e8b470b4f83ad84609f60d02573 P 700a76bf74bd4c54bf201448dee49658: No bootstrap required, opened a new log
I20260806 00:57:54.584859 25597 ts_tablet_manager.cc:1397] T 3fcb8e8b470b4f83ad84609f60d02573 P 700a76bf74bd4c54bf201448dee49658: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 00:57:54.584898 25599 tablet_bootstrap.cc:492] T 3fcb8e8b470b4f83ad84609f60d02573 P d884f2f770024d05b6c6108a691a8e23: No bootstrap required, opened a new log
I20260806 00:57:54.584990 25598 tablet_bootstrap.cc:492] T 3fcb8e8b470b4f83ad84609f60d02573 P 120c99c8c0224870a82735cd7ceba2b7: No bootstrap required, opened a new log
I20260806 00:57:54.584998 25599 ts_tablet_manager.cc:1397] T 3fcb8e8b470b4f83ad84609f60d02573 P d884f2f770024d05b6c6108a691a8e23: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 00:57:54.585057 25598 ts_tablet_manager.cc:1397] T 3fcb8e8b470b4f83ad84609f60d02573 P 120c99c8c0224870a82735cd7ceba2b7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 00:57:54.587044 25597 raft_consensus.cc:348] T 3fcb8e8b470b4f83ad84609f60d02573 P 700a76bf74bd4c54bf201448dee49658 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d884f2f770024d05b6c6108a691a8e23" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 32863 } } peers { permanent_uuid: "120c99c8c0224870a82735cd7ceba2b7" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42337 } } peers { permanent_uuid: "700a76bf74bd4c54bf201448dee49658" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37267 } }
I20260806 00:57:54.587435 25597 raft_consensus.cc:374] T 3fcb8e8b470b4f83ad84609f60d02573 P 700a76bf74bd4c54bf201448dee49658 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:57:54.587503 25597 raft_consensus.cc:729] T 3fcb8e8b470b4f83ad84609f60d02573 P 700a76bf74bd4c54bf201448dee49658 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 700a76bf74bd4c54bf201448dee49658, State: Initialized, Role: FOLLOWER
I20260806 00:57:54.587397 25599 raft_consensus.cc:348] T 3fcb8e8b470b4f83ad84609f60d02573 P d884f2f770024d05b6c6108a691a8e23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d884f2f770024d05b6c6108a691a8e23" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 32863 } } peers { permanent_uuid: "120c99c8c0224870a82735cd7ceba2b7" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42337 } } peers { permanent_uuid: "700a76bf74bd4c54bf201448dee49658" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37267 } }
I20260806 00:57:54.587397 25598 raft_consensus.cc:348] T 3fcb8e8b470b4f83ad84609f60d02573 P 120c99c8c0224870a82735cd7ceba2b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d884f2f770024d05b6c6108a691a8e23" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 32863 } } peers { permanent_uuid: "120c99c8c0224870a82735cd7ceba2b7" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42337 } } peers { permanent_uuid: "700a76bf74bd4c54bf201448dee49658" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37267 } }
I20260806 00:57:54.587708 25599 raft_consensus.cc:374] T 3fcb8e8b470b4f83ad84609f60d02573 P d884f2f770024d05b6c6108a691a8e23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:57:54.587739 25598 raft_consensus.cc:374] T 3fcb8e8b470b4f83ad84609f60d02573 P 120c99c8c0224870a82735cd7ceba2b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:57:54.587692 25597 consensus_queue.cc:260] T 3fcb8e8b470b4f83ad84609f60d02573 P 700a76bf74bd4c54bf201448dee49658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d884f2f770024d05b6c6108a691a8e23" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 32863 } } peers { permanent_uuid: "120c99c8c0224870a82735cd7ceba2b7" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42337 } } peers { permanent_uuid: "700a76bf74bd4c54bf201448dee49658" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37267 } }
I20260806 00:57:54.587790 25598 raft_consensus.cc:729] T 3fcb8e8b470b4f83ad84609f60d02573 P 120c99c8c0224870a82735cd7ceba2b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 120c99c8c0224870a82735cd7ceba2b7, State: Initialized, Role: FOLLOWER
I20260806 00:57:54.587791 25599 raft_consensus.cc:729] T 3fcb8e8b470b4f83ad84609f60d02573 P d884f2f770024d05b6c6108a691a8e23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d884f2f770024d05b6c6108a691a8e23, State: Initialized, Role: FOLLOWER
I20260806 00:57:54.587946 25599 consensus_queue.cc:260] T 3fcb8e8b470b4f83ad84609f60d02573 P d884f2f770024d05b6c6108a691a8e23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d884f2f770024d05b6c6108a691a8e23" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 32863 } } peers { permanent_uuid: "120c99c8c0224870a82735cd7ceba2b7" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42337 } } peers { permanent_uuid: "700a76bf74bd4c54bf201448dee49658" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37267 } }
I20260806 00:57:54.587946 25598 consensus_queue.cc:260] T 3fcb8e8b470b4f83ad84609f60d02573 P 120c99c8c0224870a82735cd7ceba2b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d884f2f770024d05b6c6108a691a8e23" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 32863 } } peers { permanent_uuid: "120c99c8c0224870a82735cd7ceba2b7" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42337 } } peers { permanent_uuid: "700a76bf74bd4c54bf201448dee49658" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37267 } }
I20260806 00:57:54.588071 25597 ts_tablet_manager.cc:1428] T 3fcb8e8b470b4f83ad84609f60d02573 P 700a76bf74bd4c54bf201448dee49658: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260806 00:57:54.588212 25443 heartbeater.cc:503] Master 127.23.242.62:44739 was elected leader, sending a full tablet report...
I20260806 00:57:54.588382 25598 ts_tablet_manager.cc:1428] T 3fcb8e8b470b4f83ad84609f60d02573 P 120c99c8c0224870a82735cd7ceba2b7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 00:57:54.588548 25599 ts_tablet_manager.cc:1428] T 3fcb8e8b470b4f83ad84609f60d02573 P d884f2f770024d05b6c6108a691a8e23: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260806 00:57:54.588689 25580 heartbeater.cc:503] Master 127.23.242.62:44739 was elected leader, sending a full tablet report...
I20260806 00:57:54.589406 25307 heartbeater.cc:503] Master 127.23.242.62:44739 was elected leader, sending a full tablet report...
W20260806 00:57:54.625191 25308 tablet.cc:2369] T 3fcb8e8b470b4f83ad84609f60d02573 P 120c99c8c0224870a82735cd7ceba2b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 00:57:54.756031 25604 raft_consensus.cc:482] T 3fcb8e8b470b4f83ad84609f60d02573 P d884f2f770024d05b6c6108a691a8e23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 00:57:54.756233 25604 raft_consensus.cc:504] T 3fcb8e8b470b4f83ad84609f60d02573 P d884f2f770024d05b6c6108a691a8e23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d884f2f770024d05b6c6108a691a8e23" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 32863 } } peers { permanent_uuid: "120c99c8c0224870a82735cd7ceba2b7" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42337 } } peers { permanent_uuid: "700a76bf74bd4c54bf201448dee49658" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37267 } }
I20260806 00:57:54.756675 25604 leader_election.cc:288] T 3fcb8e8b470b4f83ad84609f60d02573 P d884f2f770024d05b6c6108a691a8e23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 120c99c8c0224870a82735cd7ceba2b7 (127.23.242.1:42337), 700a76bf74bd4c54bf201448dee49658 (127.23.242.2:37267)
I20260806 00:57:54.761003 25606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:54.756950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:42337 (local address 127.23.242.3:49831)
0806 00:57:54.757466 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:54.757477 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:54.757547 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:54.757917 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:54.757918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:54.757921 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:57:54.757994 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:54.757995 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.758802 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.758803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:54.759948 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:54.759951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.760714 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.760715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:54.760794 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:54.760800 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 00:57:54.760862 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 00:57:54.760904 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":200,"thread_start_us":147,"threads_started":1}
I20260806 00:57:54.761006 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:54.757043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:49831 (local address 127.23.242.1:42337)
0806 00:57:54.757121 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:54.757124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:54.757744 (+ 620us) server_negotiation.cc:407] Connection header received
0806 00:57:54.757775 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:54.757776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:54.757785 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:54.757849 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:57:54.758094 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.758733 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.760080 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.760629 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.760715 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:54.760725 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:54.760910 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 00:57:54.760950 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 00:57:54.761432 25582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:54.756948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:37267 (local address 127.23.242.3:42563)
0806 00:57:54.757013 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:54.757026 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:54.757073 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:54.757429 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:54.757431 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:54.757434 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:57:54.757515 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:54.757517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.758567 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.758568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:54.759948 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:54.759951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.761131 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.761132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:54.761227 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:54.761232 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 00:57:54.761323 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 00:57:54.761372 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260806 00:57:54.761461 25595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:54.757081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:42563 (local address 127.23.242.2:37267)
0806 00:57:54.757137 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:54.757139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:54.757149 (+ 10us) server_negotiation.cc:407] Connection header received
0806 00:57:54.757180 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:54.757181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:54.757192 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:54.757257 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:57:54.757718 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.758470 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.760063 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.760983 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.761058 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:54.761068 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:54.761352 (+ 284us) server_negotiation.cc:299] Negotiation successful
0806 00:57:54.761400 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260806 00:57:54.761704 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fcb8e8b470b4f83ad84609f60d02573" candidate_uuid: "d884f2f770024d05b6c6108a691a8e23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "120c99c8c0224870a82735cd7ceba2b7" is_pre_election: true
I20260806 00:57:54.761888 25259 raft_consensus.cc:2454] T 3fcb8e8b470b4f83ad84609f60d02573 P 120c99c8c0224870a82735cd7ceba2b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d884f2f770024d05b6c6108a691a8e23 in term 0.
I20260806 00:57:54.761861 25397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fcb8e8b470b4f83ad84609f60d02573" candidate_uuid: "d884f2f770024d05b6c6108a691a8e23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "700a76bf74bd4c54bf201448dee49658" is_pre_election: true
I20260806 00:57:54.762022 25397 raft_consensus.cc:2454] T 3fcb8e8b470b4f83ad84609f60d02573 P 700a76bf74bd4c54bf201448dee49658 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d884f2f770024d05b6c6108a691a8e23 in term 0.
I20260806 00:57:54.762151 25465 leader_election.cc:302] T 3fcb8e8b470b4f83ad84609f60d02573 P d884f2f770024d05b6c6108a691a8e23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 120c99c8c0224870a82735cd7ceba2b7, d884f2f770024d05b6c6108a691a8e23; no voters:
I20260806 00:57:54.762338 25604 raft_consensus.cc:2781] T 3fcb8e8b470b4f83ad84609f60d02573 P d884f2f770024d05b6c6108a691a8e23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 00:57:54.762406 25604 raft_consensus.cc:482] T 3fcb8e8b470b4f83ad84609f60d02573 P d884f2f770024d05b6c6108a691a8e23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 00:57:54.762423 25604 raft_consensus.cc:3037] T 3fcb8e8b470b4f83ad84609f60d02573 P d884f2f770024d05b6c6108a691a8e23 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:57:54.763320 25604 raft_consensus.cc:504] T 3fcb8e8b470b4f83ad84609f60d02573 P d884f2f770024d05b6c6108a691a8e23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d884f2f770024d05b6c6108a691a8e23" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 32863 } } peers { permanent_uuid: "120c99c8c0224870a82735cd7ceba2b7" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42337 } } peers { permanent_uuid: "700a76bf74bd4c54bf201448dee49658" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37267 } }
I20260806 00:57:54.763500 25604 leader_election.cc:288] T 3fcb8e8b470b4f83ad84609f60d02573 P d884f2f770024d05b6c6108a691a8e23 [CANDIDATE]: Term 1 election: Requested vote from peers 120c99c8c0224870a82735cd7ceba2b7 (127.23.242.1:42337), 700a76bf74bd4c54bf201448dee49658 (127.23.242.2:37267)
I20260806 00:57:54.763672 25397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fcb8e8b470b4f83ad84609f60d02573" candidate_uuid: "d884f2f770024d05b6c6108a691a8e23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "700a76bf74bd4c54bf201448dee49658"
I20260806 00:57:54.763770 25397 raft_consensus.cc:3037] T 3fcb8e8b470b4f83ad84609f60d02573 P 700a76bf74bd4c54bf201448dee49658 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:57:54.763890 25259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fcb8e8b470b4f83ad84609f60d02573" candidate_uuid: "d884f2f770024d05b6c6108a691a8e23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "120c99c8c0224870a82735cd7ceba2b7"
I20260806 00:57:54.764146 25259 raft_consensus.cc:3037] T 3fcb8e8b470b4f83ad84609f60d02573 P 120c99c8c0224870a82735cd7ceba2b7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:57:54.764673 25397 raft_consensus.cc:2454] T 3fcb8e8b470b4f83ad84609f60d02573 P 700a76bf74bd4c54bf201448dee49658 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d884f2f770024d05b6c6108a691a8e23 in term 1.
I20260806 00:57:54.764865 25466 leader_election.cc:302] T 3fcb8e8b470b4f83ad84609f60d02573 P d884f2f770024d05b6c6108a691a8e23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 700a76bf74bd4c54bf201448dee49658, d884f2f770024d05b6c6108a691a8e23; no voters:
I20260806 00:57:54.764997 25604 raft_consensus.cc:2781] T 3fcb8e8b470b4f83ad84609f60d02573 P d884f2f770024d05b6c6108a691a8e23 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:57:54.765164 25604 raft_consensus.cc:686] T 3fcb8e8b470b4f83ad84609f60d02573 P d884f2f770024d05b6c6108a691a8e23 [term 1 LEADER]: Becoming Leader. State: Replica: d884f2f770024d05b6c6108a691a8e23, State: Running, Role: LEADER
I20260806 00:57:54.765285 25604 consensus_queue.cc:237] T 3fcb8e8b470b4f83ad84609f60d02573 P d884f2f770024d05b6c6108a691a8e23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d884f2f770024d05b6c6108a691a8e23" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 32863 } } peers { permanent_uuid: "120c99c8c0224870a82735cd7ceba2b7" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42337 } } peers { permanent_uuid: "700a76bf74bd4c54bf201448dee49658" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37267 } }
I20260806 00:57:54.765477 25259 raft_consensus.cc:2454] T 3fcb8e8b470b4f83ad84609f60d02573 P 120c99c8c0224870a82735cd7ceba2b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d884f2f770024d05b6c6108a691a8e23 in term 1.
I20260806 00:57:54.766422 25111 catalog_manager.cc:5495] T 3fcb8e8b470b4f83ad84609f60d02573 P d884f2f770024d05b6c6108a691a8e23 reported cstate change: term changed from 0 to 1, leader changed from <none> to d884f2f770024d05b6c6108a691a8e23 (127.23.242.3). New cstate: current_term: 1 leader_uuid: "d884f2f770024d05b6c6108a691a8e23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d884f2f770024d05b6c6108a691a8e23" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 32863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "120c99c8c0224870a82735cd7ceba2b7" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "700a76bf74bd4c54bf201448dee49658" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37267 } health_report { overall_health: UNKNOWN } } }
W20260806 00:57:54.781314 25581 tablet.cc:2369] T 3fcb8e8b470b4f83ad84609f60d02573 P d884f2f770024d05b6c6108a691a8e23: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 00:57:54 dist-test-slave-dhph krb5kdc[25063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.1@KRBTEST.COM
I20260806 00:57:54.816275 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:54.806880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51460 (local address 127.23.242.1:42337)
0806 00:57:54.806952 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:54.806954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:54.806964 (+ 10us) server_negotiation.cc:407] Connection header received
0806 00:57:54.806999 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:54.807000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:54.807009 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:54.807060 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:54.807881 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.808387 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.809266 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.809473 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.809549 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:54.812511 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:54.812525 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:54.812526 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:54.812657 (+ 131us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:54.814651 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:54.815180 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:54.815183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:54.815183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:54.815269 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:54.815616 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:54.815617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:54.815617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:54.815783 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:54.815910 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:54.816124 (+ 214us) server_negotiation.cc:299] Negotiation successful
0806 00:57:54.816201 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260806 00:57:54.823378 25444 tablet.cc:2369] T 3fcb8e8b470b4f83ad84609f60d02573 P 700a76bf74bd4c54bf201448dee49658: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 00:57:54 dist-test-slave-dhph krb5kdc[25063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977874, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 00:57:54 dist-test-slave-dhph krb5kdc[25063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977874, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.242.1@KRBTEST.COM
I20260806 00:57:54.847998 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:54.837038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51470 (local address 127.23.242.1:42337)
0806 00:57:54.837233 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:54.837235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:54.837328 (+ 93us) server_negotiation.cc:407] Connection header received
0806 00:57:54.837451 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:54.837452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:54.837463 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:54.837539 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:54.838244 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.838783 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.839517 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.839709 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.839788 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:54.843328 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:54.843329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:54.843330 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:54.843347 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:54.845895 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:54.846422 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:54.846423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:54.846424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:54.846496 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:54.846785 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:54.846786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:54.846787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:54.846920 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:54.847071 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:54.847569 (+ 498us) server_negotiation.cc:299] Negotiation successful
0806 00:57:54.847729 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
Aug 06 00:57:54 dist-test-slave-dhph krb5kdc[25063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 00:57:54 dist-test-slave-dhph krb5kdc[25063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.1@KRBTEST.COM
I20260806 00:57:54.879565 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:54.869805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51476 (local address 127.23.242.1:42337)
0806 00:57:54.869909 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:54.869912 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:54.869942 (+ 30us) server_negotiation.cc:407] Connection header received
0806 00:57:54.869976 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:54.869977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:54.869989 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:54.870075 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:54.870840 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.871560 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.872560 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:54.872890 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:54.872972 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:54.876328 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:54.876330 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:54.876330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:54.876339 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:54.878106 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:54.878674 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:54.878676 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:54.878676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:54.878734 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:54.878985 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:54.878986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:54.878986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:54.879150 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:54.879232 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:54.879420 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 00:57:54.879497 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260806 00:57:54.880178 25219 tablet_service.cc:2908] Rejecting scan request for tablet 3fcb8e8b470b4f83ad84609f60d02573: Uninitialized: clean time has not yet been initialized
I20260806 00:57:54.881258 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 25176
I20260806 00:57:54.888231 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 25312
I20260806 00:57:54.895296 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 25448
I20260806 00:57:54.902079 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 25076
2026-08-06T00:57:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2542 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 00:57:54 dist-test-slave-dhph krb5kdc[25621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 00:57:54 dist-test-slave-dhph krb5kdc[25621](info): set up 2 sockets
Aug 06 00:57:54 dist-test-slave-dhph krb5kdc[25621](info): commencing operation
krb5kdc: starting...
W20260806 00:57:56.083539 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 00:57:56 dist-test-slave-dhph krb5kdc[25621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T00:57:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T00:57:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 00:57:56.251541 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:39189
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:39189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:57:56.351558 25636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:57:56.351807 25636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:57:56.351841 25636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:57:56.353466 25636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 00:57:56.353543 25636 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 00:57:56.353559 25636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:57:56.353574 25636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:57:56.353587 25636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 00:57:56.353600 25636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 00:57:56.355451 25636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:39189
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:39189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:57:56.355722 25636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:57:56.355918 25636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:57:56.361339 25641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:57:56.361409 25644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:57:56.361347 25642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:57:56.361799 25636 server_base.cc:1034] running on GCE node
I20260806 00:57:56.362138 25636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:57:56.362476 25636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:57:56.363657 25636 hybrid_clock.cc:648] HybridClock initialized: now 1785977876363616 us; error 61 us; skew 500 ppm
Aug 06 00:57:56 dist-test-slave-dhph krb5kdc[25621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977876, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:57:56.366864 25636 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 00:57:56.367481 25636 webserver.cc:460] Webserver started at http://127.23.242.62:44247/ using document root <none> and password file <none>
I20260806 00:57:56.368003 25636 fs_manager.cc:359] Metadata directory not provided
I20260806 00:57:56.368062 25636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:57:56.368175 25636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:57:56.369269 25636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "7d05de3d237a44a2bedf639ef6dbd449"
format_stamp: "Formatted at 2026-08-06 00:57:56 on dist-test-slave-dhph"
server_key: "35bcdf4bc6daae8605f8ff21a94985cb"
server_key_iv: "241d41f411aa5ad149cdfe40dbff377b"
server_key_version: "encryptionkey@0"
I20260806 00:57:56.369590 25636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "7d05de3d237a44a2bedf639ef6dbd449"
format_stamp: "Formatted at 2026-08-06 00:57:56 on dist-test-slave-dhph"
server_key: "35bcdf4bc6daae8605f8ff21a94985cb"
server_key_iv: "241d41f411aa5ad149cdfe40dbff377b"
server_key_version: "encryptionkey@0"
I20260806 00:57:56.371258 25636 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 00:57:56.372684 25651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:56.373203 25636 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260806 00:57:56.373426 25636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "7d05de3d237a44a2bedf639ef6dbd449"
format_stamp: "Formatted at 2026-08-06 00:57:56 on dist-test-slave-dhph"
server_key: "35bcdf4bc6daae8605f8ff21a94985cb"
server_key_iv: "241d41f411aa5ad149cdfe40dbff377b"
server_key_version: "encryptionkey@0"
I20260806 00:57:56.373538 25636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:57:56.401782 25636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:57:56.405453 25636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:57:56.405635 25636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:57:56.411322 25636 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:39189
I20260806 00:57:56.411427 25716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:39189 every 8 connection(s)
I20260806 00:57:56.411808 25636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 00:57:56.412585 25717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:57:56.415654 25717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d05de3d237a44a2bedf639ef6dbd449: Bootstrap starting.
I20260806 00:57:56.416301 25717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d05de3d237a44a2bedf639ef6dbd449: Neither blocks nor log segments found. Creating new log.
I20260806 00:57:56.416585 25717 log.cc:824] T 00000000000000000000000000000000 P 7d05de3d237a44a2bedf639ef6dbd449: Log is configured to *not* fsync() on all Append() calls
I20260806 00:57:56.417404 25717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d05de3d237a44a2bedf639ef6dbd449: No bootstrap required, opened a new log
I20260806 00:57:56.418833 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 25636
I20260806 00:57:56.418951 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 00:57:56.419142 24520 external_mini_cluster.cc:1428] Setting key 1f96f561ecf084ac2fd2d50b8363afe1
I20260806 00:57:56.420619 25717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d05de3d237a44a2bedf639ef6dbd449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d05de3d237a44a2bedf639ef6dbd449" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39189 } }
I20260806 00:57:56.420810 25717 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d05de3d237a44a2bedf639ef6dbd449 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:57:56.420850 25717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d05de3d237a44a2bedf639ef6dbd449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d05de3d237a44a2bedf639ef6dbd449, State: Initialized, Role: FOLLOWER
I20260806 00:57:56.420992 25717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d05de3d237a44a2bedf639ef6dbd449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d05de3d237a44a2bedf639ef6dbd449" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39189 } }
I20260806 00:57:56.421096 25717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d05de3d237a44a2bedf639ef6dbd449 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 00:57:56.421136 25717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d05de3d237a44a2bedf639ef6dbd449 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 00:57:56.421181 25717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d05de3d237a44a2bedf639ef6dbd449 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:57:56.422945 25717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d05de3d237a44a2bedf639ef6dbd449 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d05de3d237a44a2bedf639ef6dbd449" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39189 } }
I20260806 00:57:56.423069 25717 leader_election.cc:302] T 00000000000000000000000000000000 P 7d05de3d237a44a2bedf639ef6dbd449 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d05de3d237a44a2bedf639ef6dbd449; no voters:
I20260806 00:57:56.423661 25717 leader_election.cc:288] T 00000000000000000000000000000000 P 7d05de3d237a44a2bedf639ef6dbd449 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 00:57:56.423704 25722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d05de3d237a44a2bedf639ef6dbd449 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:57:56.423854 25722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d05de3d237a44a2bedf639ef6dbd449 [term 1 LEADER]: Becoming Leader. State: Replica: 7d05de3d237a44a2bedf639ef6dbd449, State: Running, Role: LEADER
I20260806 00:57:56.423956 25722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d05de3d237a44a2bedf639ef6dbd449 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d05de3d237a44a2bedf639ef6dbd449" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39189 } }
I20260806 00:57:56.424031 25717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d05de3d237a44a2bedf639ef6dbd449 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 00:57:56.425083 25724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d05de3d237a44a2bedf639ef6dbd449 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d05de3d237a44a2bedf639ef6dbd449. Latest consensus state: current_term: 1 leader_uuid: "7d05de3d237a44a2bedf639ef6dbd449" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d05de3d237a44a2bedf639ef6dbd449" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39189 } } }
I20260806 00:57:56.425201 25724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d05de3d237a44a2bedf639ef6dbd449 [sys.catalog]: This master's current role is: LEADER
I20260806 00:57:56.425505 25723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d05de3d237a44a2bedf639ef6dbd449 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d05de3d237a44a2bedf639ef6dbd449" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d05de3d237a44a2bedf639ef6dbd449" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39189 } } }
I20260806 00:57:56.425757 25723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d05de3d237a44a2bedf639ef6dbd449 [sys.catalog]: This master's current role is: LEADER
I20260806 00:57:56.426867 25728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 00:57:56.427543 25728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 06 00:57:56 dist-test-slave-dhph krb5kdc[25621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:57:56.430343 25728 catalog_manager.cc:1347] Generated new cluster ID: eeb2c80e13e4494f8b2a7a5bcdd2116a
I20260806 00:57:56.431855 25728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 00:57:56.434077 25721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:56.419897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47218 (local address 127.23.242.62:39189)
0806 00:57:56.420132 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:56.420142 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:56.420177 (+ 35us) server_negotiation.cc:407] Connection header received
0806 00:57:56.420598 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:56.420608 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:56.420664 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:56.420834 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:56.421794 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:56.423180 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:56.424333 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:56.425066 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:56.425169 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:56.429062 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:56.429066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:56.429066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:56.429081 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:56.431587 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:56.432426 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:56.432429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:56.432429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:56.432575 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:56.432849 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:56.432850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:56.432850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:56.433264 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:56.433475 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:56.433555 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 00:57:56.433716 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":122,"threads_started":1}
I20260806 00:57:56.453049 25728 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 00:57:56.453639 25728 catalog_manager.cc:1504] Loading token signing keys...
I20260806 00:57:56.460573 25728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d05de3d237a44a2bedf639ef6dbd449: Generated new TSK 0
I20260806 00:57:56.460819 25728 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.242.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
I20260806 00:57:56.514948 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:0
--local_ip_for_outbound_sockets=127.23.242.1
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:39189
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:57:56.673143 25743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:57:56.673477 25743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:57:56.673518 25743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:57:56.675475 25743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:57:56.675695 25743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:57:56.675758 25743 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 00:57:56.675792 25743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 00:57:56.678318 25743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.242.62:39189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:57:56.678558 25743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:57:56.678869 25743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:57:56.684909 25749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:57:56.685001 25748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:57:56.685141 25743 server_base.cc:1034] running on GCE node
W20260806 00:57:56.684901 25751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:57:56.685539 25743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:57:56.685845 25743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:57:56.687079 25743 hybrid_clock.cc:648] HybridClock initialized: now 1785977876687061 us; error 32 us; skew 500 ppm
Aug 06 00:57:56 dist-test-slave-dhph krb5kdc[25621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977876, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:57:56.690399 25743 init.cc:377] Logged in from keytab as kudu/127.23.242.1@KRBTEST.COM (short username kudu)
I20260806 00:57:56.690946 25743 webserver.cc:460] Webserver started at http://127.23.242.1:34021/ using document root <none> and password file <none>
I20260806 00:57:56.691197 25743 fs_manager.cc:359] Metadata directory not provided
I20260806 00:57:56.691246 25743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:57:56.691370 25743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:57:56.692519 25743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-0/data/instance:
uuid: "900b93eac37a4cb68d442d4ef7859989"
format_stamp: "Formatted at 2026-08-06 00:57:56 on dist-test-slave-dhph"
server_key: "dfb0c7eb12834a08d376741b331ac75c"
server_key_iv: "047e049b91cafa683a9a110016414776"
server_key_version: "encryptionkey@0"
I20260806 00:57:56.692881 25743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance:
uuid: "900b93eac37a4cb68d442d4ef7859989"
format_stamp: "Formatted at 2026-08-06 00:57:56 on dist-test-slave-dhph"
server_key: "dfb0c7eb12834a08d376741b331ac75c"
server_key_iv: "047e049b91cafa683a9a110016414776"
server_key_version: "encryptionkey@0"
I20260806 00:57:56.695225 25743 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260806 00:57:56.696307 25758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:56.696873 25743 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 00:57:56.697007 25743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "900b93eac37a4cb68d442d4ef7859989"
format_stamp: "Formatted at 2026-08-06 00:57:56 on dist-test-slave-dhph"
server_key: "dfb0c7eb12834a08d376741b331ac75c"
server_key_iv: "047e049b91cafa683a9a110016414776"
server_key_version: "encryptionkey@0"
I20260806 00:57:56.697124 25743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:57:56.710322 25743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:57:56.713392 25743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:57:56.713537 25743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:57:56.714180 25743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:57:56.714260 25743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:56.714289 25743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:57:56.714569 25743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:56.722729 25743 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:46733
Aug 06 00:57:56 dist-test-slave-dhph krb5kdc[25621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977876, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:57:56.722844 25876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:46733 every 8 connection(s)
I20260806 00:57:56.723240 25743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 00:57:56.727983 25721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:56.715353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:36443 (local address 127.23.242.62:39189)
0806 00:57:56.715424 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:56.715426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:56.715647 (+ 221us) server_negotiation.cc:407] Connection header received
0806 00:57:56.716246 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:56.716247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:56.716265 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:56.716333 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:56.717290 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:56.718175 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:56.719989 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:56.720384 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:56.720478 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:56.723755 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:56.723757 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:56.723758 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:56.723774 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:56.725772 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:56.726331 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:56.726332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:56.726332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:56.726403 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:56.727170 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:56.727171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:56.727172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:56.727315 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:56.727393 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:56.727809 (+ 416us) server_negotiation.cc:299] Negotiation successful
0806 00:57:56.727903 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 00:57:56.728106 25768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:56.715307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:39189 (local address 127.23.242.1:36443)
0806 00:57:56.715516 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:56.715555 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:56.716126 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:56.716419 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:56.716423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:56.716838 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:57:56.717153 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:56.717160 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:56.718298 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:56.718299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:56.719890 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:56.719893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:56.720514 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:56.720515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:56.720654 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:56.721309 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:57:56.721321 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:57:56.723525 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:57:56.725909 (+ 2384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:56.725917 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:56.725919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:56.726217 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:56.726665 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:56.726667 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:56.726668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:56.727055 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:56.727422 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:57:56.727429 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:57:56.727578 (+ 149us) client_negotiation.cc:769] Sending connection context
0806 00:57:56.727680 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 00:57:56.727859 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":115,"threads_started":1}
I20260806 00:57:56.728554 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 25743
I20260806 00:57:56.728677 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance
I20260806 00:57:56.728884 24520 external_mini_cluster.cc:1428] Setting key f59aedc138a96022f95c5e311930ed76
I20260806 00:57:56.729079 25877 heartbeater.cc:347] Connected to a master server at 127.23.242.62:39189
I20260806 00:57:56.729179 25877 heartbeater.cc:464] Registering TS with master...
I20260806 00:57:56.729388 25877 heartbeater.cc:511] Master 127.23.242.62:39189 requested a full tablet report, sending...
I20260806 00:57:56.730965 25670 ts_manager.cc:194] Registered new tserver with Master: 900b93eac37a4cb68d442d4ef7859989 (127.23.242.1:46733)
I20260806 00:57:56.732007 25670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.1@KRBTEST.COM'} at 127.23.242.1:36443
I20260806 00:57:56.737484 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:56.730110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:39189 (local address 127.23.242.1:42467)
0806 00:57:56.730319 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:56.730329 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:56.730383 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:56.730513 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:56.730514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:56.730755 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:57:56.730837 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:56.730839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:56.732097 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:56.732098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:56.732799 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:56.732803 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:56.733311 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:56.733312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:56.733376 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:56.733903 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:57:56.733908 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:57:56.734252 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:57:56.736238 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:56.736239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:56.736239 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:56.736455 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:56.736698 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:56.736699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:56.736699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:56.736764 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:56.737099 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:57:56.737100 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:57:56.737155 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 00:57:56.737237 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 00:57:56.737392 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":67,"threads_started":1}
I20260806 00:57:56.737499 25721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:56.730185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:42467 (local address 127.23.242.62:39189)
0806 00:57:56.730238 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:56.730240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:56.730394 (+ 154us) server_negotiation.cc:407] Connection header received
0806 00:57:56.730424 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:56.730425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:56.730438 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:56.730512 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:56.730938 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:56.731816 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:56.732917 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:56.733241 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:56.733312 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:56.734359 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:56.734360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:56.734361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:56.734369 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:56.736111 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:56.736541 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:56.736542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:56.736543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:56.736613 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:56.736839 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:56.736841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:56.736841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:56.736994 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:56.737074 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:56.737289 (+ 215us) server_negotiation.cc:299] Negotiation successful
0806 00:57:56.737394 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":24}
WARNING: no policy specified for kudu/127.23.242.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
I20260806 00:57:56.768534 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:0
--local_ip_for_outbound_sockets=127.23.242.2
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:39189
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:57:56.873010 25887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:57:56.873291 25887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:57:56.873342 25887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:57:56.875365 25887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:57:56.875476 25887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:57:56.875526 25887 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 00:57:56.875597 25887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 00:57:56.877784 25887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.242.62:39189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:57:56.878120 25887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:57:56.878399 25887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:57:56.884716 25892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:57:56.884698 25896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:57:56.884698 25893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:57:56.885006 25887 server_base.cc:1034] running on GCE node
I20260806 00:57:56.885350 25887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:57:56.885648 25887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:57:56.886790 25887 hybrid_clock.cc:648] HybridClock initialized: now 1785977876886780 us; error 35 us; skew 500 ppm
Aug 06 00:57:56 dist-test-slave-dhph krb5kdc[25621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977876, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:57:56.889940 25887 init.cc:377] Logged in from keytab as kudu/127.23.242.2@KRBTEST.COM (short username kudu)
I20260806 00:57:56.890544 25887 webserver.cc:460] Webserver started at http://127.23.242.2:38151/ using document root <none> and password file <none>
I20260806 00:57:56.890869 25887 fs_manager.cc:359] Metadata directory not provided
I20260806 00:57:56.890928 25887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:57:56.891034 25887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:57:56.892611 25887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-1/data/instance:
uuid: "75fdb5ac10a04820802de6a57786370e"
format_stamp: "Formatted at 2026-08-06 00:57:56 on dist-test-slave-dhph"
server_key: "bed9d8e8605dbd557593f10af6d79dd4"
server_key_iv: "5370f1935abd63ab170f082b14b2c692"
server_key_version: "encryptionkey@0"
I20260806 00:57:56.893174 25887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance:
uuid: "75fdb5ac10a04820802de6a57786370e"
format_stamp: "Formatted at 2026-08-06 00:57:56 on dist-test-slave-dhph"
server_key: "bed9d8e8605dbd557593f10af6d79dd4"
server_key_iv: "5370f1935abd63ab170f082b14b2c692"
server_key_version: "encryptionkey@0"
I20260806 00:57:56.895750 25887 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.003s
I20260806 00:57:56.897045 25903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:56.897475 25887 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:57:56.897639 25887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "75fdb5ac10a04820802de6a57786370e"
format_stamp: "Formatted at 2026-08-06 00:57:56 on dist-test-slave-dhph"
server_key: "bed9d8e8605dbd557593f10af6d79dd4"
server_key_iv: "5370f1935abd63ab170f082b14b2c692"
server_key_version: "encryptionkey@0"
I20260806 00:57:56.897989 25887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:57:56.911007 25887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:57:56.916056 25887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:57:56.916601 25887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:57:56.918072 25887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:57:56.918175 25887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:56.918213 25887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:57:56.918241 25887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 00:57:56 dist-test-slave-dhph krb5kdc[25621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977876, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:57:56.932992 25721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:56.918663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:49303 (local address 127.23.242.62:39189)
0806 00:57:56.918738 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:56.918741 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:56.919083 (+ 342us) server_negotiation.cc:407] Connection header received
0806 00:57:56.919620 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:56.919622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:56.919638 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:56.919737 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:56.920614 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:56.921688 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:56.923481 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:56.923760 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:56.923861 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:56.927571 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:56.927573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:56.927573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:56.927583 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:56.930323 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:56.931505 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:56.931508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:56.931508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:56.931618 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:56.932123 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:56.932124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:56.932125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:56.932285 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:56.932467 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:56.932793 (+ 326us) server_negotiation.cc:299] Negotiation successful
0806 00:57:56.932897 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 00:57:56.933218 25912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:56.918636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:39189 (local address 127.23.242.2:49303)
0806 00:57:56.918839 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:56.918882 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:56.919367 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:56.919990 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:56.919994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:56.920258 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:57:56.920459 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:56.920463 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:56.921861 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:56.921863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:56.923026 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:56.923030 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:56.923884 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:56.923885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:56.924060 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:56.924813 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:57:56.924824 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:57:56.927427 (+ 2603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:57:56.930485 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:56.930503 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:56.930506 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:56.931363 (+ 857us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:56.931743 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:56.931745 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:56.931745 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:56.931978 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:56.932454 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:57:56.932462 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:57:56.932632 (+ 170us) client_negotiation.cc:769] Sending connection context
0806 00:57:56.932750 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 00:57:56.932940 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":118,"threads_started":1}
I20260806 00:57:56.933305 25887 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:36389
I20260806 00:57:56.933887 25887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 00:57:56.934332 26020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:36389 every 8 connection(s)
I20260806 00:57:56.934504 26021 heartbeater.cc:347] Connected to a master server at 127.23.242.62:39189
I20260806 00:57:56.934628 26021 heartbeater.cc:464] Registering TS with master...
I20260806 00:57:56.934804 26021 heartbeater.cc:511] Master 127.23.242.62:39189 requested a full tablet report, sending...
I20260806 00:57:56.935519 25670 ts_manager.cc:194] Registered new tserver with Master: 75fdb5ac10a04820802de6a57786370e (127.23.242.2:36389)
I20260806 00:57:56.936197 25670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.2@KRBTEST.COM'} at 127.23.242.2:49303
I20260806 00:57:56.936430 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 25887
I20260806 00:57:56.936537 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance
I20260806 00:57:56.936730 24520 external_mini_cluster.cc:1428] Setting key 94f3f2c24a77977f5fb9db20dcfdb7fe
I20260806 00:57:56.945956 25721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:56.935767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:53675 (local address 127.23.242.62:39189)
0806 00:57:56.935922 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:56.935924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:56.935963 (+ 39us) server_negotiation.cc:407] Connection header received
0806 00:57:56.936038 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:56.936039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:56.936048 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:56.936106 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:56.936699 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:56.937926 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:56.938991 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:56.939258 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:56.939343 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:56.941318 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:56.941322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:56.941323 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:56.941339 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:56.943494 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:56.944268 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:56.944270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:56.944271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:56.944373 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:56.944789 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:56.944790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:56.944791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:56.945035 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:56.945116 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:56.945782 (+ 666us) server_negotiation.cc:299] Negotiation successful
0806 00:57:56.945877 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 00:57:56.946009 26027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:56.935549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:39189 (local address 127.23.242.2:53675)
0806 00:57:56.935882 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:56.935896 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:56.935983 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:56.936119 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:56.936120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:56.936471 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:57:56.936621 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:56.936624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:56.938081 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:56.938083 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:56.938870 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:56.938874 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:56.939364 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:56.939366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:56.939463 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:56.940025 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:57:56.940030 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:57:56.940449 (+ 419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:57:56.943781 (+ 3332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:56.943783 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:56.943783 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:56.944112 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:56.944492 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:56.944493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:56.944493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:56.944583 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:56.945278 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:57:56.945280 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:57:56.945348 (+ 68us) client_negotiation.cc:769] Sending connection context
0806 00:57:56.945692 (+ 344us) client_negotiation.cc:240] Negotiation successful
0806 00:57:56.945854 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":196,"threads_started":1}
WARNING: no policy specified for kudu/127.23.242.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
I20260806 00:57:56.975513 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:0
--local_ip_for_outbound_sockets=127.23.242.3
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:39189
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:57:57.073518 26030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:57:57.073767 26030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:57:57.073796 26030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:57:57.075119 26030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:57:57.075186 26030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:57:57.075210 26030 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 00:57:57.075229 26030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 00:57:57.077003 26030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.242.62:39189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:57:57.077375 26030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:57:57.077694 26030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:57:57.084107 26036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:57:57.084107 26035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:57:57.084195 26038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:57:57.084728 26030 server_base.cc:1034] running on GCE node
I20260806 00:57:57.084955 26030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:57:57.085321 26030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:57:57.086505 26030 hybrid_clock.cc:648] HybridClock initialized: now 1785977877086456 us; error 69 us; skew 500 ppm
Aug 06 00:57:57 dist-test-slave-dhph krb5kdc[25621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977877, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:57:57.089160 26030 init.cc:377] Logged in from keytab as kudu/127.23.242.3@KRBTEST.COM (short username kudu)
I20260806 00:57:57.089656 26030 webserver.cc:460] Webserver started at http://127.23.242.3:39671/ using document root <none> and password file <none>
I20260806 00:57:57.089857 26030 fs_manager.cc:359] Metadata directory not provided
I20260806 00:57:57.089900 26030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:57:57.090003 26030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:57:57.091143 26030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-2/data/instance:
uuid: "3e926b5bc3984f908ed53d24b091adb8"
format_stamp: "Formatted at 2026-08-06 00:57:57 on dist-test-slave-dhph"
server_key: "fbf69e3aaf8cd0ffb501cd73d8ed26de"
server_key_iv: "0a4c5caa3bd87b58bf6bc9d532da4940"
server_key_version: "encryptionkey@0"
I20260806 00:57:57.091598 26030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance:
uuid: "3e926b5bc3984f908ed53d24b091adb8"
format_stamp: "Formatted at 2026-08-06 00:57:57 on dist-test-slave-dhph"
server_key: "fbf69e3aaf8cd0ffb501cd73d8ed26de"
server_key_iv: "0a4c5caa3bd87b58bf6bc9d532da4940"
server_key_version: "encryptionkey@0"
I20260806 00:57:57.093381 26030 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 00:57:57.094516 26045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:57.094776 26030 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:57:57.094877 26030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "3e926b5bc3984f908ed53d24b091adb8"
format_stamp: "Formatted at 2026-08-06 00:57:57 on dist-test-slave-dhph"
server_key: "fbf69e3aaf8cd0ffb501cd73d8ed26de"
server_key_iv: "0a4c5caa3bd87b58bf6bc9d532da4940"
server_key_version: "encryptionkey@0"
I20260806 00:57:57.094951 26030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:57:57.105732 26030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:57:57.108326 26030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:57:57.108456 26030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:57:57.109001 26030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:57:57.109050 26030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:57.109076 26030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:57:57.109090 26030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:57.117537 26030 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:42861
I20260806 00:57:57.117635 26165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:42861 every 8 connection(s)
I20260806 00:57:57.118006 26030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
Aug 06 00:57:57 dist-test-slave-dhph krb5kdc[25621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977877, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:57:57.123030 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 26030
I20260806 00:57:57.123082 25721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:57.109895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:35093 (local address 127.23.242.62:39189)
0806 00:57:57.110031 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:57.110034 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:57.110459 (+ 425us) server_negotiation.cc:407] Connection header received
0806 00:57:57.111447 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:57.111448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:57.111465 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:57.111523 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:57.112601 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.113661 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.115266 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.115670 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.115766 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:57.118942 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:57.118944 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:57.118944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:57.118953 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:57.120870 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:57.121546 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:57.121549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:57.121549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:57.121676 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:57.122125 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:57.122126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:57.122127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:57.122280 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:57.122352 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:57.122848 (+ 496us) server_negotiation.cc:299] Negotiation successful
0806 00:57:57.122961 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 00:57:57.123139 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:57.110013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:39189 (local address 127.23.242.3:35093)
0806 00:57:57.110299 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:57.110351 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:57.111312 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:57.111610 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:57.111615 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:57.112066 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:57:57.112474 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:57.112483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.113780 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.113781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:57.115150 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:57.115153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.115832 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.115834 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:57.116160 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:57.116863 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:57:57.116872 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:57:57.118806 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:57:57.121006 (+ 2200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:57.121016 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:57.121018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:57.121318 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:57.121825 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:57.121825 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:57.121826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:57.121960 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:57.122382 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:57:57.122389 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:57:57.122538 (+ 149us) client_negotiation.cc:769] Sending connection context
0806 00:57:57.122742 (+ 204us) client_negotiation.cc:240] Negotiation successful
0806 00:57:57.122918 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":145,"threads_started":1}
I20260806 00:57:57.123159 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance
I20260806 00:57:57.123397 24520 external_mini_cluster.cc:1428] Setting key d1dcb41085a6fad59f2be759f2c70cf4
I20260806 00:57:57.124135 26166 heartbeater.cc:347] Connected to a master server at 127.23.242.62:39189
I20260806 00:57:57.124246 26166 heartbeater.cc:464] Registering TS with master...
I20260806 00:57:57.124498 26166 heartbeater.cc:511] Master 127.23.242.62:39189 requested a full tablet report, sending...
I20260806 00:57:57.125159 25668 ts_manager.cc:194] Registered new tserver with Master: 3e926b5bc3984f908ed53d24b091adb8 (127.23.242.3:42861)
I20260806 00:57:57.125696 25668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.3@KRBTEST.COM'} at 127.23.242.3:35093
I20260806 00:57:57.133059 25721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:57.125869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:35519 (local address 127.23.242.62:39189)
0806 00:57:57.125922 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:57.125925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:57.126171 (+ 246us) server_negotiation.cc:407] Connection header received
0806 00:57:57.126199 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:57.126200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:57.126212 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:57.126282 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:57.126741 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.127295 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.128130 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.128354 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.128439 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:57.129718 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:57.129719 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:57.129720 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:57.129728 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:57.131427 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:57.131920 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:57.131922 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:57.131922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:57.131992 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:57.132357 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:57.132358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:57.132359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:57.132513 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:57.132602 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:57.132846 (+ 244us) server_negotiation.cc:299] Negotiation successful
0806 00:57:57.132946 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 00:57:57.133075 26172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:57.125841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:39189 (local address 127.23.242.3:35519)
0806 00:57:57.126057 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:57.126068 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:57.126134 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:57.126286 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:57.126287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:57.126504 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:57:57.126633 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:57.126636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.127401 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.127402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:57.128008 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:57.128010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.128438 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.128439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:57.128531 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:57.129203 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:57:57.129208 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:57:57.129599 (+ 391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:57:57.131557 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:57.131558 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:57.131559 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:57.131826 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:57.132079 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:57.132079 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:57.132080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:57.132153 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:57.132606 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:57:57.132607 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:57:57.132666 (+ 59us) client_negotiation.cc:769] Sending connection context
0806 00:57:57.132814 (+ 148us) client_negotiation.cc:240] Negotiation successful
0806 00:57:57.132947 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":141,"threads_started":1}
I20260806 00:57:57.135939 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 00:57:57 dist-test-slave-dhph krb5kdc[25621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977877, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 00:57:57 dist-test-slave-dhph krb5kdc[25621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977877, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:57:57.168109 25721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:57.158290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47222 (local address 127.23.242.62:39189)
0806 00:57:57.158379 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:57.158382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:57.158392 (+ 10us) server_negotiation.cc:407] Connection header received
0806 00:57:57.158421 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:57.158422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:57.158432 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:57.158495 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:57.159488 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.160069 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.161085 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.161320 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.161395 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:57.164551 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:57.164553 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:57.164553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:57.164561 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:57.166123 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:57.166731 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:57.166734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:57.166734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:57.166866 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:57.167302 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:57.167303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:57.167304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:57.167469 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:57.167627 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:57.167944 (+ 317us) server_negotiation.cc:299] Negotiation successful
0806 00:57:57.168025 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 00:57:57.170531 25668 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 00:57:57.172715 25668 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 00:57:57.180541 26183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:57.175700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:42861 (local address 127.0.0.1:41782)
0806 00:57:57.175924 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:57.175972 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:57.176053 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:57.176738 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:57.176739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:57.176741 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:57:57.176781 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:57.176782 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.178240 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.178241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:57.179269 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:57.179272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.180181 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.180182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:57.180286 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:57.180302 (+ 16us) client_negotiation.cc:769] Sending connection context
0806 00:57:57.180386 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 00:57:57.180439 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":72,"thread_start_us":57,"threads_started":1}
I20260806 00:57:57.180581 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:57.176050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41782 (local address 127.23.242.3:42861)
0806 00:57:57.176283 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:57.176385 (+ 102us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:57.176403 (+ 18us) server_negotiation.cc:407] Connection header received
0806 00:57:57.176508 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:57.176511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:57.176530 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:57.176717 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:57:57.176878 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.178141 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.179376 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.180079 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.180165 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:57.180229 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:57.180421 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 00:57:57.180469 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":100,"threads_started":1}
I20260806 00:57:57.181047 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:57.175709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:46733 (local address 127.0.0.1:55386)
0806 00:57:57.176291 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:57.176303 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:57.176362 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:57.176595 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:57.176598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:57.176600 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:57:57.176690 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:57.176692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.178198 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.178199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:57.179655 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:57.179658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.180760 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.180761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:57.180842 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:57.180848 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 00:57:57.180913 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 00:57:57.180968 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":109,"threads_started":1}
I20260806 00:57:57.181413 26185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:57.175844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55386 (local address 127.23.242.1:46733)
0806 00:57:57.176295 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:57.176302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:57.176352 (+ 50us) server_negotiation.cc:407] Connection header received
0806 00:57:57.176440 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:57.176443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:57.176464 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:57.176554 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:57:57.176786 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.178112 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.179759 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.180556 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.180650 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:57.180717 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:57.181253 (+ 536us) server_negotiation.cc:299] Negotiation successful
0806 00:57:57.181307 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":99,"threads_started":1}
I20260806 00:57:57.181615 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:57.175914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:36389 (local address 127.0.0.1:32888)
0806 00:57:57.176453 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:57.176462 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:57.176504 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:57.177018 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:57.177019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:57.177022 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:57:57.177076 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:57.177078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.178115 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.178116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:57.179877 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:57.179879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.181319 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.181321 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:57.181396 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:57.181401 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 00:57:57.181480 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 00:57:57.181538 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":80,"thread_start_us":83,"threads_started":1}
I20260806 00:57:57.181664 26098 tablet_service.cc:1467] Processing CreateTablet for tablet a196a234dc134474a7610a0ecd119404 (DEFAULT_TABLE table=test-table [id=dde40a4f2b1c4b3bb061db2ec82b4e31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:57:57.181970 26187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:57.176025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32888 (local address 127.23.242.2:36389)
0806 00:57:57.176857 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:57.176862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:57.176872 (+ 10us) server_negotiation.cc:407] Connection header received
0806 00:57:57.176913 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:57.176915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:57.176934 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:57.177020 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:57:57.177178 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.178017 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.179968 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.181130 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.181222 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:57.181282 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:57.181642 (+ 360us) server_negotiation.cc:299] Negotiation successful
0806 00:57:57.181837 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":140,"threads_started":1}
I20260806 00:57:57.182128 26098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a196a234dc134474a7610a0ecd119404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:57:57.182020 25810 tablet_service.cc:1467] Processing CreateTablet for tablet a196a234dc134474a7610a0ecd119404 (DEFAULT_TABLE table=test-table [id=dde40a4f2b1c4b3bb061db2ec82b4e31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:57:57.182377 25810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a196a234dc134474a7610a0ecd119404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:57:57.183300 25955 tablet_service.cc:1467] Processing CreateTablet for tablet a196a234dc134474a7610a0ecd119404 (DEFAULT_TABLE table=test-table [id=dde40a4f2b1c4b3bb061db2ec82b4e31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:57:57.183652 25955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a196a234dc134474a7610a0ecd119404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:57:57.187543 26189 tablet_bootstrap.cc:492] T a196a234dc134474a7610a0ecd119404 P 75fdb5ac10a04820802de6a57786370e: Bootstrap starting.
I20260806 00:57:57.187918 26190 tablet_bootstrap.cc:492] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8: Bootstrap starting.
I20260806 00:57:57.188436 26191 tablet_bootstrap.cc:492] T a196a234dc134474a7610a0ecd119404 P 900b93eac37a4cb68d442d4ef7859989: Bootstrap starting.
I20260806 00:57:57.188639 26189 tablet_bootstrap.cc:654] T a196a234dc134474a7610a0ecd119404 P 75fdb5ac10a04820802de6a57786370e: Neither blocks nor log segments found. Creating new log.
I20260806 00:57:57.189023 26190 tablet_bootstrap.cc:654] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8: Neither blocks nor log segments found. Creating new log.
I20260806 00:57:57.189144 26191 tablet_bootstrap.cc:654] T a196a234dc134474a7610a0ecd119404 P 900b93eac37a4cb68d442d4ef7859989: Neither blocks nor log segments found. Creating new log.
I20260806 00:57:57.189086 26189 log.cc:824] T a196a234dc134474a7610a0ecd119404 P 75fdb5ac10a04820802de6a57786370e: Log is configured to *not* fsync() on all Append() calls
I20260806 00:57:57.189424 26190 log.cc:824] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8: Log is configured to *not* fsync() on all Append() calls
I20260806 00:57:57.189491 26191 log.cc:824] T a196a234dc134474a7610a0ecd119404 P 900b93eac37a4cb68d442d4ef7859989: Log is configured to *not* fsync() on all Append() calls
I20260806 00:57:57.190187 26189 tablet_bootstrap.cc:492] T a196a234dc134474a7610a0ecd119404 P 75fdb5ac10a04820802de6a57786370e: No bootstrap required, opened a new log
I20260806 00:57:57.190231 26190 tablet_bootstrap.cc:492] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8: No bootstrap required, opened a new log
I20260806 00:57:57.190270 26189 ts_tablet_manager.cc:1397] T a196a234dc134474a7610a0ecd119404 P 75fdb5ac10a04820802de6a57786370e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 00:57:57.190321 26190 ts_tablet_manager.cc:1397] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 00:57:57.190373 26191 tablet_bootstrap.cc:492] T a196a234dc134474a7610a0ecd119404 P 900b93eac37a4cb68d442d4ef7859989: No bootstrap required, opened a new log
I20260806 00:57:57.190424 26191 ts_tablet_manager.cc:1397] T a196a234dc134474a7610a0ecd119404 P 900b93eac37a4cb68d442d4ef7859989: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 00:57:57.192165 26190 raft_consensus.cc:348] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } } peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } }
I20260806 00:57:57.192346 26190 raft_consensus.cc:374] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:57:57.192397 26190 raft_consensus.cc:729] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e926b5bc3984f908ed53d24b091adb8, State: Initialized, Role: FOLLOWER
I20260806 00:57:57.192523 26190 consensus_queue.cc:260] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } } peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } }
I20260806 00:57:57.192831 26190 ts_tablet_manager.cc:1428] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 00:57:57.192838 26166 heartbeater.cc:503] Master 127.23.242.62:39189 was elected leader, sending a full tablet report...
I20260806 00:57:57.193200 26189 raft_consensus.cc:348] T a196a234dc134474a7610a0ecd119404 P 75fdb5ac10a04820802de6a57786370e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } } peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } }
I20260806 00:57:57.193390 26191 raft_consensus.cc:348] T a196a234dc134474a7610a0ecd119404 P 900b93eac37a4cb68d442d4ef7859989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } } peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } }
I20260806 00:57:57.193555 26191 raft_consensus.cc:374] T a196a234dc134474a7610a0ecd119404 P 900b93eac37a4cb68d442d4ef7859989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:57:57.193605 26191 raft_consensus.cc:729] T a196a234dc134474a7610a0ecd119404 P 900b93eac37a4cb68d442d4ef7859989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900b93eac37a4cb68d442d4ef7859989, State: Initialized, Role: FOLLOWER
I20260806 00:57:57.193688 26189 raft_consensus.cc:374] T a196a234dc134474a7610a0ecd119404 P 75fdb5ac10a04820802de6a57786370e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:57:57.193765 26189 raft_consensus.cc:729] T a196a234dc134474a7610a0ecd119404 P 75fdb5ac10a04820802de6a57786370e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75fdb5ac10a04820802de6a57786370e, State: Initialized, Role: FOLLOWER
I20260806 00:57:57.193818 26191 consensus_queue.cc:260] T a196a234dc134474a7610a0ecd119404 P 900b93eac37a4cb68d442d4ef7859989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } } peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } }
I20260806 00:57:57.193924 26189 consensus_queue.cc:260] T a196a234dc134474a7610a0ecd119404 P 75fdb5ac10a04820802de6a57786370e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } } peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } }
I20260806 00:57:57.194147 26191 ts_tablet_manager.cc:1428] T a196a234dc134474a7610a0ecd119404 P 900b93eac37a4cb68d442d4ef7859989: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260806 00:57:57.194238 26189 ts_tablet_manager.cc:1428] T a196a234dc134474a7610a0ecd119404 P 75fdb5ac10a04820802de6a57786370e: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260806 00:57:57.194396 26021 heartbeater.cc:503] Master 127.23.242.62:39189 was elected leader, sending a full tablet report...
I20260806 00:57:57.194490 25877 heartbeater.cc:503] Master 127.23.242.62:39189 was elected leader, sending a full tablet report...
I20260806 00:57:57.194794 26196 raft_consensus.cc:482] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 00:57:57.194881 26196 raft_consensus.cc:504] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } } peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } }
I20260806 00:57:57.195245 26196 leader_election.cc:288] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 900b93eac37a4cb68d442d4ef7859989 (127.23.242.1:46733), 75fdb5ac10a04820802de6a57786370e (127.23.242.2:36389)
I20260806 00:57:57.198932 26199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:57.195391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:46733 (local address 127.23.242.3:47407)
0806 00:57:57.195834 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:57.195844 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:57.195910 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:57.196025 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:57.196026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:57.196027 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:57:57.196100 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:57.196102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.196816 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.196817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:57.197774 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:57.197777 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.198572 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.198612 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:57.198701 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:57.198706 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 00:57:57.198797 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 00:57:57.198841 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":246,"thread_start_us":149,"threads_started":1}
I20260806 00:57:57.198918 26185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:57.195610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:47407 (local address 127.23.242.1:46733)
0806 00:57:57.195653 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:57.195655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:57.195915 (+ 260us) server_negotiation.cc:407] Connection header received
0806 00:57:57.195964 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:57.195965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:57.195974 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:57.196025 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:57:57.196188 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.196708 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.197888 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.198479 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.198565 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:57.198609 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:57.198808 (+ 199us) server_negotiation.cc:299] Negotiation successful
0806 00:57:57.198849 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 00:57:57.199298 25830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a196a234dc134474a7610a0ecd119404" candidate_uuid: "3e926b5bc3984f908ed53d24b091adb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900b93eac37a4cb68d442d4ef7859989" is_pre_election: true
I20260806 00:57:57.199486 25830 raft_consensus.cc:2454] T a196a234dc134474a7610a0ecd119404 P 900b93eac37a4cb68d442d4ef7859989 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e926b5bc3984f908ed53d24b091adb8 in term 0.
I20260806 00:57:57.199765 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:57.195391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:36389 (local address 127.23.242.3:50337)
0806 00:57:57.195476 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:57.195489 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:57.195531 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:57.195870 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:57.195872 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:57.195875 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:57:57.195950 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:57.195957 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.196717 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.196718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:57.198264 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:57.198267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.199365 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.199366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:57.199481 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:57.199557 (+ 76us) client_negotiation.cc:769] Sending connection context
0806 00:57:57.199638 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 00:57:57.199687 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 00:57:57.200019 26187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:57.195557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:50337 (local address 127.23.242.2:36389)
0806 00:57:57.195710 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:57.195712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:57.195723 (+ 11us) server_negotiation.cc:407] Connection header received
0806 00:57:57.195745 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:57.195746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:57.195754 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:57.195848 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:57:57.196037 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.196618 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.198379 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.199255 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.199352 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:57.199362 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:57.199889 (+ 527us) server_negotiation.cc:299] Negotiation successful
0806 00:57:57.199943 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 00:57:57.200031 26049 leader_election.cc:302] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e926b5bc3984f908ed53d24b091adb8, 900b93eac37a4cb68d442d4ef7859989; no voters:
I20260806 00:57:57.200227 26196 raft_consensus.cc:2781] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 00:57:57.200280 26196 raft_consensus.cc:482] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 00:57:57.200336 26196 raft_consensus.cc:3037] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:57:57.200444 25975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a196a234dc134474a7610a0ecd119404" candidate_uuid: "3e926b5bc3984f908ed53d24b091adb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75fdb5ac10a04820802de6a57786370e" is_pre_election: true
I20260806 00:57:57.200609 25975 raft_consensus.cc:2454] T a196a234dc134474a7610a0ecd119404 P 75fdb5ac10a04820802de6a57786370e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e926b5bc3984f908ed53d24b091adb8 in term 0.
I20260806 00:57:57.201295 26196 raft_consensus.cc:504] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } } peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } }
I20260806 00:57:57.201432 26196 leader_election.cc:288] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8 [CANDIDATE]: Term 1 election: Requested vote from peers 900b93eac37a4cb68d442d4ef7859989 (127.23.242.1:46733), 75fdb5ac10a04820802de6a57786370e (127.23.242.2:36389)
I20260806 00:57:57.201647 25975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a196a234dc134474a7610a0ecd119404" candidate_uuid: "3e926b5bc3984f908ed53d24b091adb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75fdb5ac10a04820802de6a57786370e"
I20260806 00:57:57.201651 25830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a196a234dc134474a7610a0ecd119404" candidate_uuid: "3e926b5bc3984f908ed53d24b091adb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900b93eac37a4cb68d442d4ef7859989"
I20260806 00:57:57.201738 25975 raft_consensus.cc:3037] T a196a234dc134474a7610a0ecd119404 P 75fdb5ac10a04820802de6a57786370e [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:57:57.201763 25830 raft_consensus.cc:3037] T a196a234dc134474a7610a0ecd119404 P 900b93eac37a4cb68d442d4ef7859989 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:57:57.202534 25975 raft_consensus.cc:2454] T a196a234dc134474a7610a0ecd119404 P 75fdb5ac10a04820802de6a57786370e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e926b5bc3984f908ed53d24b091adb8 in term 1.
I20260806 00:57:57.202666 25830 raft_consensus.cc:2454] T a196a234dc134474a7610a0ecd119404 P 900b93eac37a4cb68d442d4ef7859989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e926b5bc3984f908ed53d24b091adb8 in term 1.
I20260806 00:57:57.203433 26047 leader_election.cc:302] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e926b5bc3984f908ed53d24b091adb8, 75fdb5ac10a04820802de6a57786370e; no voters:
I20260806 00:57:57.203711 26196 raft_consensus.cc:2781] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:57:57.204042 26196 raft_consensus.cc:686] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8 [term 1 LEADER]: Becoming Leader. State: Replica: 3e926b5bc3984f908ed53d24b091adb8, State: Running, Role: LEADER
I20260806 00:57:57.204195 26196 consensus_queue.cc:237] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } } peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } }
I20260806 00:57:57.205278 25670 catalog_manager.cc:5495] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e926b5bc3984f908ed53d24b091adb8 (127.23.242.3). New cstate: current_term: 1 leader_uuid: "3e926b5bc3984f908ed53d24b091adb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } health_report { overall_health: UNKNOWN } } }
I20260806 00:57:57.222502 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:57.218596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41794 (local address 127.23.242.3:42861)
0806 00:57:57.218739 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:57.218742 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:57.218780 (+ 38us) server_negotiation.cc:407] Connection header received
0806 00:57:57.218942 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:57.218943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:57.218964 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:57.219032 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 00:57:57.219294 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.219947 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.221217 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.221411 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.221479 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:57.221809 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 00:57:57.221920 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 00:57:57.222003 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:57.222247 (+ 244us) server_negotiation.cc:299] Negotiation successful
0806 00:57:57.222297 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260806 00:57:57.224995 25975 raft_consensus.cc:1261] T a196a234dc134474a7610a0ecd119404 P 75fdb5ac10a04820802de6a57786370e [term 1 FOLLOWER]: Refusing update from remote peer 3e926b5bc3984f908ed53d24b091adb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 00:57:57.224960 25878 tablet.cc:2369] T a196a234dc134474a7610a0ecd119404 P 900b93eac37a4cb68d442d4ef7859989: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 00:57:57.225174 25830 raft_consensus.cc:1261] T a196a234dc134474a7610a0ecd119404 P 900b93eac37a4cb68d442d4ef7859989 [term 1 FOLLOWER]: Refusing update from remote peer 3e926b5bc3984f908ed53d24b091adb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:57:57.225426 26196 consensus_queue.cc:1035] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:57:57.225652 26200 consensus_queue.cc:1035] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:57:57.228669 26203 mvcc.cc:205] Tried to move back new op lower bound from 7315365385109925888 to 7315365385029148672. Current Snapshot: MvccSnapshot[applied={T|T < 7315365385109925888}]
I20260806 00:57:57.228858 26205 mvcc.cc:205] Tried to move back new op lower bound from 7315365385109925888 to 7315365385029148672. Current Snapshot: MvccSnapshot[applied={T|T < 7315365385109925888}]
I20260806 00:57:57.233973 25668 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:57:57.234086 25668 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:57:57.235335 25668 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7d05de3d237a44a2bedf639ef6dbd449: Sending DeleteTablet for 3 replicas of tablet a196a234dc134474a7610a0ecd119404
I20260806 00:57:57.235941 26098 tablet_service.cc:1514] Processing DeleteTablet for tablet a196a234dc134474a7610a0ecd119404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:57:57 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:41782
I20260806 00:57:57.236037 25810 tablet_service.cc:1514] Processing DeleteTablet for tablet a196a234dc134474a7610a0ecd119404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:57:57 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:55386
I20260806 00:57:57.236121 25955 tablet_service.cc:1514] Processing DeleteTablet for tablet a196a234dc134474a7610a0ecd119404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:57:57 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:32888
I20260806 00:57:57.236403 26214 tablet_replica.cc:330] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8: stopping tablet replica
I20260806 00:57:57.236406 26215 tablet_replica.cc:330] T a196a234dc134474a7610a0ecd119404 P 900b93eac37a4cb68d442d4ef7859989: stopping tablet replica
I20260806 00:57:57.236531 26214 raft_consensus.cc:2229] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8 [term 1 LEADER]: Raft consensus shutting down.
I20260806 00:57:57.236683 26214 raft_consensus.cc:2258] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:57:57.236588 25668 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 00:57:57.236869 26216 tablet_replica.cc:330] T a196a234dc134474a7610a0ecd119404 P 75fdb5ac10a04820802de6a57786370e: stopping tablet replica
W20260806 00:57:57.236965 25668 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 00:57:57.236984 26216 raft_consensus.cc:2229] T a196a234dc134474a7610a0ecd119404 P 75fdb5ac10a04820802de6a57786370e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:57:57.237136 26215 raft_consensus.cc:2229] T a196a234dc134474a7610a0ecd119404 P 900b93eac37a4cb68d442d4ef7859989 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:57:57.237315 26215 raft_consensus.cc:2258] T a196a234dc134474a7610a0ecd119404 P 900b93eac37a4cb68d442d4ef7859989 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:57:57.237102 26216 raft_consensus.cc:2258] T a196a234dc134474a7610a0ecd119404 P 75fdb5ac10a04820802de6a57786370e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:57:57.238260 26215 ts_tablet_manager.cc:1905] T a196a234dc134474a7610a0ecd119404 P 900b93eac37a4cb68d442d4ef7859989: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:57:57.238740 25810 tablet_service.cc:1467] Processing CreateTablet for tablet a515a68831b1496496a390d2ba877a76 (DEFAULT_TABLE table=test-table [id=2baf9dba5dbc4be0b997ac674666667c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:57:57.238847 26216 ts_tablet_manager.cc:1905] T a196a234dc134474a7610a0ecd119404 P 75fdb5ac10a04820802de6a57786370e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:57:57.238893 25810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a515a68831b1496496a390d2ba877a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:57:57.239070 25955 tablet_service.cc:1467] Processing CreateTablet for tablet a515a68831b1496496a390d2ba877a76 (DEFAULT_TABLE table=test-table [id=2baf9dba5dbc4be0b997ac674666667c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:57:57.239188 25955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a515a68831b1496496a390d2ba877a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:57:57.240330 26191 tablet_bootstrap.cc:492] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989: Bootstrap starting.
I20260806 00:57:57.240419 26098 tablet_service.cc:1467] Processing CreateTablet for tablet a515a68831b1496496a390d2ba877a76 (DEFAULT_TABLE table=test-table [id=2baf9dba5dbc4be0b997ac674666667c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:57:57.240548 26098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a515a68831b1496496a390d2ba877a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:57:57.240686 26189 tablet_bootstrap.cc:492] T a515a68831b1496496a390d2ba877a76 P 75fdb5ac10a04820802de6a57786370e: Bootstrap starting.
I20260806 00:57:57.240769 26191 tablet_bootstrap.cc:654] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989: Neither blocks nor log segments found. Creating new log.
I20260806 00:57:57.241133 26189 tablet_bootstrap.cc:654] T a515a68831b1496496a390d2ba877a76 P 75fdb5ac10a04820802de6a57786370e: Neither blocks nor log segments found. Creating new log.
I20260806 00:57:57.238652 26214 ts_tablet_manager.cc:1905] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:57:57.241931 26189 tablet_bootstrap.cc:492] T a515a68831b1496496a390d2ba877a76 P 75fdb5ac10a04820802de6a57786370e: No bootstrap required, opened a new log
I20260806 00:57:57.241993 26189 ts_tablet_manager.cc:1397] T a515a68831b1496496a390d2ba877a76 P 75fdb5ac10a04820802de6a57786370e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260806 00:57:57.242348 26190 tablet_bootstrap.cc:492] T a515a68831b1496496a390d2ba877a76 P 3e926b5bc3984f908ed53d24b091adb8: Bootstrap starting.
I20260806 00:57:57.242287 26189 raft_consensus.cc:348] T a515a68831b1496496a390d2ba877a76 P 75fdb5ac10a04820802de6a57786370e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } }
I20260806 00:57:57.242404 26189 raft_consensus.cc:374] T a515a68831b1496496a390d2ba877a76 P 75fdb5ac10a04820802de6a57786370e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:57:57.242435 26189 raft_consensus.cc:729] T a515a68831b1496496a390d2ba877a76 P 75fdb5ac10a04820802de6a57786370e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75fdb5ac10a04820802de6a57786370e, State: Initialized, Role: FOLLOWER
I20260806 00:57:57.242492 26189 consensus_queue.cc:260] T a515a68831b1496496a390d2ba877a76 P 75fdb5ac10a04820802de6a57786370e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } }
I20260806 00:57:57.242906 26190 tablet_bootstrap.cc:654] T a515a68831b1496496a390d2ba877a76 P 3e926b5bc3984f908ed53d24b091adb8: Neither blocks nor log segments found. Creating new log.
I20260806 00:57:57.243357 26216 ts_tablet_manager.cc:1918] T a196a234dc134474a7610a0ecd119404 P 75fdb5ac10a04820802de6a57786370e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 00:57:57.243431 26216 log.cc:1196] T a196a234dc134474a7610a0ecd119404 P 75fdb5ac10a04820802de6a57786370e: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-1/wal/wals/a196a234dc134474a7610a0ecd119404
I20260806 00:57:57.243721 26216 ts_tablet_manager.cc:1939] T a196a234dc134474a7610a0ecd119404 P 75fdb5ac10a04820802de6a57786370e: Deleting consensus metadata
I20260806 00:57:57.244195 25654 catalog_manager.cc:4841] TS 75fdb5ac10a04820802de6a57786370e (127.23.242.2:36389): tablet a196a234dc134474a7610a0ecd119404 (table test-table [id=dde40a4f2b1c4b3bb061db2ec82b4e31]) successfully deleted
I20260806 00:57:57.244227 26214 ts_tablet_manager.cc:1918] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 00:57:57.244299 26214 log.cc:1196] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-2/wal/wals/a196a234dc134474a7610a0ecd119404
I20260806 00:57:57.244557 26214 ts_tablet_manager.cc:1939] T a196a234dc134474a7610a0ecd119404 P 3e926b5bc3984f908ed53d24b091adb8: Deleting consensus metadata
I20260806 00:57:57.244565 26189 ts_tablet_manager.cc:1428] T a515a68831b1496496a390d2ba877a76 P 75fdb5ac10a04820802de6a57786370e: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260806 00:57:57.245455 26215 ts_tablet_manager.cc:1918] T a196a234dc134474a7610a0ecd119404 P 900b93eac37a4cb68d442d4ef7859989: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 00:57:57.245584 26215 log.cc:1196] T a196a234dc134474a7610a0ecd119404 P 900b93eac37a4cb68d442d4ef7859989: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-0/wal/wals/a196a234dc134474a7610a0ecd119404
I20260806 00:57:57.245582 25654 catalog_manager.cc:4841] TS 3e926b5bc3984f908ed53d24b091adb8 (127.23.242.3:42861): tablet a196a234dc134474a7610a0ecd119404 (table test-table [id=dde40a4f2b1c4b3bb061db2ec82b4e31]) successfully deleted
I20260806 00:57:57.245860 26215 ts_tablet_manager.cc:1939] T a196a234dc134474a7610a0ecd119404 P 900b93eac37a4cb68d442d4ef7859989: Deleting consensus metadata
I20260806 00:57:57.246385 26191 tablet_bootstrap.cc:492] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989: No bootstrap required, opened a new log
I20260806 00:57:57.246449 26191 ts_tablet_manager.cc:1397] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260806 00:57:57.246630 25656 catalog_manager.cc:4841] TS 900b93eac37a4cb68d442d4ef7859989 (127.23.242.1:46733): tablet a196a234dc134474a7610a0ecd119404 (table test-table [id=dde40a4f2b1c4b3bb061db2ec82b4e31]) successfully deleted
I20260806 00:57:57.246658 26191 raft_consensus.cc:348] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } }
I20260806 00:57:57.246747 26191 raft_consensus.cc:374] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:57:57.246770 26191 raft_consensus.cc:729] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900b93eac37a4cb68d442d4ef7859989, State: Initialized, Role: FOLLOWER
I20260806 00:57:57.246819 26191 consensus_queue.cc:260] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } }
I20260806 00:57:57.246927 26191 ts_tablet_manager.cc:1428] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260806 00:57:57.252491 26190 tablet_bootstrap.cc:492] T a515a68831b1496496a390d2ba877a76 P 3e926b5bc3984f908ed53d24b091adb8: No bootstrap required, opened a new log
I20260806 00:57:57.252559 26190 ts_tablet_manager.cc:1397] T a515a68831b1496496a390d2ba877a76 P 3e926b5bc3984f908ed53d24b091adb8: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260806 00:57:57.252794 26190 raft_consensus.cc:348] T a515a68831b1496496a390d2ba877a76 P 3e926b5bc3984f908ed53d24b091adb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } }
I20260806 00:57:57.252869 26190 raft_consensus.cc:374] T a515a68831b1496496a390d2ba877a76 P 3e926b5bc3984f908ed53d24b091adb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:57:57.252884 26190 raft_consensus.cc:729] T a515a68831b1496496a390d2ba877a76 P 3e926b5bc3984f908ed53d24b091adb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e926b5bc3984f908ed53d24b091adb8, State: Initialized, Role: FOLLOWER
I20260806 00:57:57.252941 26190 consensus_queue.cc:260] T a515a68831b1496496a390d2ba877a76 P 3e926b5bc3984f908ed53d24b091adb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } }
I20260806 00:57:57.253044 26190 ts_tablet_manager.cc:1428] T a515a68831b1496496a390d2ba877a76 P 3e926b5bc3984f908ed53d24b091adb8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260806 00:57:57.368841 26167 tablet.cc:2369] T a515a68831b1496496a390d2ba877a76 P 3e926b5bc3984f908ed53d24b091adb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 00:57:57.434870 26022 tablet.cc:2369] T a515a68831b1496496a390d2ba877a76 P 75fdb5ac10a04820802de6a57786370e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 00:57:57.574782 26197 raft_consensus.cc:482] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 00:57:57.574946 26197 raft_consensus.cc:504] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } }
I20260806 00:57:57.575826 26197 leader_election.cc:288] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75fdb5ac10a04820802de6a57786370e (127.23.242.2:36389), 3e926b5bc3984f908ed53d24b091adb8 (127.23.242.3:42861)
I20260806 00:57:57.579542 26221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:57.576202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:36389 (local address 127.23.242.1:40951)
0806 00:57:57.576352 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:57.576365 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:57.576446 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:57.576628 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:57.576629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:57.576632 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:57:57.576759 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:57.576761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.577492 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.577493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:57.578560 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:57.578564 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.579255 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.579257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:57.579333 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:57.579338 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 00:57:57.579399 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 00:57:57.579444 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":75,"threads_started":1}
I20260806 00:57:57.579594 26187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:57.576299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:40951 (local address 127.23.242.2:36389)
0806 00:57:57.576351 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:57.576353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:57.576462 (+ 109us) server_negotiation.cc:407] Connection header received
0806 00:57:57.576506 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:57.576507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:57.576518 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:57.576599 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:57:57.576816 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.577416 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.578696 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.579153 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.579234 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:57.579246 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:57.579493 (+ 247us) server_negotiation.cc:299] Negotiation successful
0806 00:57:57.579538 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 00:57:57.579887 25975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a515a68831b1496496a390d2ba877a76" candidate_uuid: "900b93eac37a4cb68d442d4ef7859989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75fdb5ac10a04820802de6a57786370e" is_pre_election: true
I20260806 00:57:57.579984 25975 raft_consensus.cc:2454] T a515a68831b1496496a390d2ba877a76 P 75fdb5ac10a04820802de6a57786370e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 900b93eac37a4cb68d442d4ef7859989 in term 0.
I20260806 00:57:57.580199 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:57.576487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:51371 (local address 127.23.242.3:42861)
0806 00:57:57.576536 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:57.576538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:57.576666 (+ 128us) server_negotiation.cc:407] Connection header received
0806 00:57:57.576756 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:57.576757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:57.576769 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:57.576819 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:57:57.577061 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.577888 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.579308 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.579845 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.579928 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:57.579936 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:57.580092 (+ 156us) server_negotiation.cc:299] Negotiation successful
0806 00:57:57.580133 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 00:57:57.580225 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:57.576403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:42861 (local address 127.23.242.1:51371)
0806 00:57:57.576584 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:57.576596 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:57.576669 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:57.576875 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:57.576876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:57.576878 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:57:57.576958 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:57.576959 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.578017 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.578018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:57.579215 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:57.579235 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.579928 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.579929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:57.580002 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:57.580006 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 00:57:57.580083 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 00:57:57.580135 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":65,"threads_started":1}
I20260806 00:57:57.580224 25761 leader_election.cc:302] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75fdb5ac10a04820802de6a57786370e, 900b93eac37a4cb68d442d4ef7859989; no voters:
I20260806 00:57:57.580440 26197 raft_consensus.cc:2781] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 00:57:57.580488 26197 raft_consensus.cc:482] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 00:57:57.580523 26197 raft_consensus.cc:3037] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:57:57.580603 26119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a515a68831b1496496a390d2ba877a76" candidate_uuid: "900b93eac37a4cb68d442d4ef7859989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e926b5bc3984f908ed53d24b091adb8" is_pre_election: true
I20260806 00:57:57.580760 26119 raft_consensus.cc:2454] T a515a68831b1496496a390d2ba877a76 P 3e926b5bc3984f908ed53d24b091adb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 900b93eac37a4cb68d442d4ef7859989 in term 0.
I20260806 00:57:57.581341 26197 raft_consensus.cc:504] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } }
I20260806 00:57:57.581494 26197 leader_election.cc:288] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989 [CANDIDATE]: Term 1 election: Requested vote from peers 75fdb5ac10a04820802de6a57786370e (127.23.242.2:36389), 3e926b5bc3984f908ed53d24b091adb8 (127.23.242.3:42861)
I20260806 00:57:57.581683 25975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a515a68831b1496496a390d2ba877a76" candidate_uuid: "900b93eac37a4cb68d442d4ef7859989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75fdb5ac10a04820802de6a57786370e"
I20260806 00:57:57.581785 25975 raft_consensus.cc:3037] T a515a68831b1496496a390d2ba877a76 P 75fdb5ac10a04820802de6a57786370e [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:57:57.581708 26119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a515a68831b1496496a390d2ba877a76" candidate_uuid: "900b93eac37a4cb68d442d4ef7859989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e926b5bc3984f908ed53d24b091adb8"
I20260806 00:57:57.581836 26119 raft_consensus.cc:3037] T a515a68831b1496496a390d2ba877a76 P 3e926b5bc3984f908ed53d24b091adb8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:57:57.582702 25975 raft_consensus.cc:2454] T a515a68831b1496496a390d2ba877a76 P 75fdb5ac10a04820802de6a57786370e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 900b93eac37a4cb68d442d4ef7859989 in term 1.
I20260806 00:57:57.582753 26119 raft_consensus.cc:2454] T a515a68831b1496496a390d2ba877a76 P 3e926b5bc3984f908ed53d24b091adb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 900b93eac37a4cb68d442d4ef7859989 in term 1.
I20260806 00:57:57.582887 25761 leader_election.cc:302] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75fdb5ac10a04820802de6a57786370e, 900b93eac37a4cb68d442d4ef7859989; no voters:
I20260806 00:57:57.583020 26197 raft_consensus.cc:2781] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:57:57.583168 26197 raft_consensus.cc:686] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989 [term 1 LEADER]: Becoming Leader. State: Replica: 900b93eac37a4cb68d442d4ef7859989, State: Running, Role: LEADER
I20260806 00:57:57.583262 26197 consensus_queue.cc:237] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } }
I20260806 00:57:57.584141 25668 catalog_manager.cc:5495] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989 reported cstate change: term changed from 0 to 1, leader changed from <none> to 900b93eac37a4cb68d442d4ef7859989 (127.23.242.1). New cstate: current_term: 1 leader_uuid: "900b93eac37a4cb68d442d4ef7859989" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } health_report { overall_health: UNKNOWN } } }
Aug 06 00:57:57 dist-test-slave-dhph krb5kdc[25621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977876, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:57:57.633752 26230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:57.623498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:39189 (local address 127.0.0.1:48316)
0806 00:57:57.623727 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:57.623749 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:57.623808 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:57.624161 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:57.624163 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:57.624570 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:57:57.624652 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:57.624655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.625602 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.625603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:57.626126 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:57.626128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.626479 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.626480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:57.626529 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:57.627290 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:57:57.627298 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:57:57.629665 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:57:57.631881 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:57.631892 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:57.631894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:57.632153 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:57.632687 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:57.632688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:57.632688 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:57.632765 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:57.633218 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:57:57.633303 (+ 85us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:57:57.633375 (+ 72us) client_negotiation.cc:769] Sending connection context
0806 00:57:57.633465 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 00:57:57.633588 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":163,"threads_started":1}
I20260806 00:57:57.633958 25721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:57.623673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48316 (local address 127.23.242.62:39189)
0806 00:57:57.623829 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:57.623832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:57.623895 (+ 63us) server_negotiation.cc:407] Connection header received
0806 00:57:57.624029 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:57.624031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:57.624043 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:57.624118 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:57.624729 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.625472 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.626212 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.626401 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.626481 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:57.629830 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:57.629831 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:57.629832 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:57.629840 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:57.631745 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:57.632308 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:57.632310 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:57.632310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:57.632524 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:57.632913 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:57.632914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:57.632915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:57.633070 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:57.633145 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:57.633575 (+ 430us) server_negotiation.cc:299] Negotiation successful
0806 00:57:57.633834 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 00:57:57.636960 25668 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:48316:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 00:57:57.637228 25668 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 00:57:57.639192 25955 tablet_service.cc:1467] Processing CreateTablet for tablet 59fa09e81a6843438a18bcbdfad6b4c3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=daef5318728146649a641ab0ee865b77]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 00:57:57.639222 26098 tablet_service.cc:1467] Processing CreateTablet for tablet 59fa09e81a6843438a18bcbdfad6b4c3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=daef5318728146649a641ab0ee865b77]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 00:57:57.639288 25810 tablet_service.cc:1467] Processing CreateTablet for tablet 59fa09e81a6843438a18bcbdfad6b4c3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=daef5318728146649a641ab0ee865b77]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 00:57:57.639395 25955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59fa09e81a6843438a18bcbdfad6b4c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:57:57.639418 26098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59fa09e81a6843438a18bcbdfad6b4c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:57:57.639424 25810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59fa09e81a6843438a18bcbdfad6b4c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:57:57.640987 26190 tablet_bootstrap.cc:492] T 59fa09e81a6843438a18bcbdfad6b4c3 P 3e926b5bc3984f908ed53d24b091adb8: Bootstrap starting.
I20260806 00:57:57.641398 26190 tablet_bootstrap.cc:654] T 59fa09e81a6843438a18bcbdfad6b4c3 P 3e926b5bc3984f908ed53d24b091adb8: Neither blocks nor log segments found. Creating new log.
I20260806 00:57:57.641673 26189 tablet_bootstrap.cc:492] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e: Bootstrap starting.
I20260806 00:57:57.642093 26189 tablet_bootstrap.cc:654] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e: Neither blocks nor log segments found. Creating new log.
I20260806 00:57:57.642160 26190 tablet_bootstrap.cc:492] T 59fa09e81a6843438a18bcbdfad6b4c3 P 3e926b5bc3984f908ed53d24b091adb8: No bootstrap required, opened a new log
I20260806 00:57:57.642223 26190 ts_tablet_manager.cc:1397] T 59fa09e81a6843438a18bcbdfad6b4c3 P 3e926b5bc3984f908ed53d24b091adb8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 00:57:57.642349 26190 raft_consensus.cc:348] T 59fa09e81a6843438a18bcbdfad6b4c3 P 3e926b5bc3984f908ed53d24b091adb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } }
I20260806 00:57:57.642463 26190 raft_consensus.cc:374] T 59fa09e81a6843438a18bcbdfad6b4c3 P 3e926b5bc3984f908ed53d24b091adb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:57:57.642462 26191 tablet_bootstrap.cc:492] T 59fa09e81a6843438a18bcbdfad6b4c3 P 900b93eac37a4cb68d442d4ef7859989: Bootstrap starting.
I20260806 00:57:57.642489 26190 raft_consensus.cc:729] T 59fa09e81a6843438a18bcbdfad6b4c3 P 3e926b5bc3984f908ed53d24b091adb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e926b5bc3984f908ed53d24b091adb8, State: Initialized, Role: FOLLOWER
I20260806 00:57:57.642549 26190 consensus_queue.cc:260] T 59fa09e81a6843438a18bcbdfad6b4c3 P 3e926b5bc3984f908ed53d24b091adb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } }
I20260806 00:57:57.642730 26190 ts_tablet_manager.cc:1428] T 59fa09e81a6843438a18bcbdfad6b4c3 P 3e926b5bc3984f908ed53d24b091adb8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260806 00:57:57.642868 26209 tablet_replica.cc:439] T 59fa09e81a6843438a18bcbdfad6b4c3 P 3e926b5bc3984f908ed53d24b091adb8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } } }
I20260806 00:57:57.642931 26191 tablet_bootstrap.cc:654] T 59fa09e81a6843438a18bcbdfad6b4c3 P 900b93eac37a4cb68d442d4ef7859989: Neither blocks nor log segments found. Creating new log.
I20260806 00:57:57.642968 26209 tablet_replica.cc:442] T 59fa09e81a6843438a18bcbdfad6b4c3 P 3e926b5bc3984f908ed53d24b091adb8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 00:57:57.643757 26191 tablet_bootstrap.cc:492] T 59fa09e81a6843438a18bcbdfad6b4c3 P 900b93eac37a4cb68d442d4ef7859989: No bootstrap required, opened a new log
I20260806 00:57:57.643780 26189 tablet_bootstrap.cc:492] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e: No bootstrap required, opened a new log
I20260806 00:57:57.643829 26189 ts_tablet_manager.cc:1397] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 00:57:57.643836 26191 ts_tablet_manager.cc:1397] T 59fa09e81a6843438a18bcbdfad6b4c3 P 900b93eac37a4cb68d442d4ef7859989: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260806 00:57:57.644006 26189 raft_consensus.cc:348] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } }
I20260806 00:57:57.644068 26189 raft_consensus.cc:374] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:57:57.644088 26189 raft_consensus.cc:729] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75fdb5ac10a04820802de6a57786370e, State: Initialized, Role: FOLLOWER
I20260806 00:57:57.644016 26191 raft_consensus.cc:348] T 59fa09e81a6843438a18bcbdfad6b4c3 P 900b93eac37a4cb68d442d4ef7859989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } }
I20260806 00:57:57.644109 26191 raft_consensus.cc:374] T 59fa09e81a6843438a18bcbdfad6b4c3 P 900b93eac37a4cb68d442d4ef7859989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:57:57.644147 26191 raft_consensus.cc:729] T 59fa09e81a6843438a18bcbdfad6b4c3 P 900b93eac37a4cb68d442d4ef7859989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900b93eac37a4cb68d442d4ef7859989, State: Initialized, Role: FOLLOWER
I20260806 00:57:57.644143 26189 consensus_queue.cc:260] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } }
I20260806 00:57:57.644212 26191 consensus_queue.cc:260] T 59fa09e81a6843438a18bcbdfad6b4c3 P 900b93eac37a4cb68d442d4ef7859989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } }
I20260806 00:57:57.644261 26189 ts_tablet_manager.cc:1428] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:57.644316 26191 ts_tablet_manager.cc:1428] T 59fa09e81a6843438a18bcbdfad6b4c3 P 900b93eac37a4cb68d442d4ef7859989: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260806 00:57:57.644317 26198 tablet_replica.cc:439] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } } }
I20260806 00:57:57.644413 26198 tablet_replica.cc:442] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 00:57:57.644505 26197 tablet_replica.cc:439] T 59fa09e81a6843438a18bcbdfad6b4c3 P 900b93eac37a4cb68d442d4ef7859989: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } } }
I20260806 00:57:57.644616 26197 tablet_replica.cc:442] T 59fa09e81a6843438a18bcbdfad6b4c3 P 900b93eac37a4cb68d442d4ef7859989: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 00:57:57.971725 26198 raft_consensus.cc:482] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 00:57:57.971859 26198 raft_consensus.cc:504] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } }
I20260806 00:57:57.972661 26198 leader_election.cc:288] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 900b93eac37a4cb68d442d4ef7859989 (127.23.242.1:46733), 3e926b5bc3984f908ed53d24b091adb8 (127.23.242.3:42861)
I20260806 00:57:57.977037 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:57.973118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:43075 (local address 127.23.242.3:42861)
0806 00:57:57.973285 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:57.973293 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:57.973453 (+ 160us) server_negotiation.cc:407] Connection header received
0806 00:57:57.973485 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:57.973486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:57.973497 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:57.973544 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:57:57.973834 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.974354 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.975659 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.976474 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.976704 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:57.976719 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:57.976898 (+ 179us) server_negotiation.cc:299] Negotiation successful
0806 00:57:57.976949 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 00:57:57.977063 26234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:57.972954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:42861 (local address 127.23.242.2:43075)
0806 00:57:57.973132 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:57.973146 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:57.973244 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:57.973586 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:57.973588 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:57.973592 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:57:57.973719 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:57.973723 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.974471 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.974473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:57.975445 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:57.975449 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.976717 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.976718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:57.976798 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:57.976803 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 00:57:57.976873 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 00:57:57.976923 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":79,"threads_started":1}
I20260806 00:57:57.977596 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:57.972869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:46733 (local address 127.23.242.2:45865)
0806 00:57:57.973100 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:57.973117 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:57.973302 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:57.973757 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:57.973759 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:57.973762 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:57:57.973868 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:57.973870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.974881 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.974882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:57.976189 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:57.976193 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.977288 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.977289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:57.977357 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:57.977360 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 00:57:57.977429 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 00:57:57.977496 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":1,"thread_start_us":109,"threads_started":1}
I20260806 00:57:57.977537 26119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fa09e81a6843438a18bcbdfad6b4c3" candidate_uuid: "75fdb5ac10a04820802de6a57786370e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e926b5bc3984f908ed53d24b091adb8" is_pre_election: true
I20260806 00:57:57.977751 26119 raft_consensus.cc:2454] T 59fa09e81a6843438a18bcbdfad6b4c3 P 3e926b5bc3984f908ed53d24b091adb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75fdb5ac10a04820802de6a57786370e in term 0.
I20260806 00:57:57.977993 26235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:57.972943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:45865 (local address 127.23.242.1:46733)
0806 00:57:57.973137 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:57.973140 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:57.973221 (+ 81us) server_negotiation.cc:407] Connection header received
0806 00:57:57.973577 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:57.973580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:57.973600 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:57.973729 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:57:57.974027 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.974732 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.976320 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:57.977156 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:57.977240 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:57.977251 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:57.977820 (+ 569us) server_negotiation.cc:299] Negotiation successful
0806 00:57:57.977894 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":87,"threads_started":1}
I20260806 00:57:57.978014 25905 leader_election.cc:302] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e926b5bc3984f908ed53d24b091adb8, 75fdb5ac10a04820802de6a57786370e; no voters:
I20260806 00:57:57.978183 26198 raft_consensus.cc:2781] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 00:57:57.978228 26198 raft_consensus.cc:482] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 00:57:57.978253 26198 raft_consensus.cc:3037] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:57:57.978338 25830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fa09e81a6843438a18bcbdfad6b4c3" candidate_uuid: "75fdb5ac10a04820802de6a57786370e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900b93eac37a4cb68d442d4ef7859989" is_pre_election: true
I20260806 00:57:57.978440 25830 raft_consensus.cc:2454] T 59fa09e81a6843438a18bcbdfad6b4c3 P 900b93eac37a4cb68d442d4ef7859989 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75fdb5ac10a04820802de6a57786370e in term 0.
I20260806 00:57:57.979154 26198 raft_consensus.cc:504] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } }
I20260806 00:57:57.979310 26198 leader_election.cc:288] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e [CANDIDATE]: Term 1 election: Requested vote from peers 900b93eac37a4cb68d442d4ef7859989 (127.23.242.1:46733), 3e926b5bc3984f908ed53d24b091adb8 (127.23.242.3:42861)
I20260806 00:57:57.979478 26119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fa09e81a6843438a18bcbdfad6b4c3" candidate_uuid: "75fdb5ac10a04820802de6a57786370e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e926b5bc3984f908ed53d24b091adb8"
I20260806 00:57:57.979741 26119 raft_consensus.cc:3037] T 59fa09e81a6843438a18bcbdfad6b4c3 P 3e926b5bc3984f908ed53d24b091adb8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:57:57.979791 25830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fa09e81a6843438a18bcbdfad6b4c3" candidate_uuid: "75fdb5ac10a04820802de6a57786370e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "900b93eac37a4cb68d442d4ef7859989"
I20260806 00:57:57.979864 25830 raft_consensus.cc:3037] T 59fa09e81a6843438a18bcbdfad6b4c3 P 900b93eac37a4cb68d442d4ef7859989 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:57:57.980638 25830 raft_consensus.cc:2454] T 59fa09e81a6843438a18bcbdfad6b4c3 P 900b93eac37a4cb68d442d4ef7859989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75fdb5ac10a04820802de6a57786370e in term 1.
I20260806 00:57:57.980636 26119 raft_consensus.cc:2454] T 59fa09e81a6843438a18bcbdfad6b4c3 P 3e926b5bc3984f908ed53d24b091adb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75fdb5ac10a04820802de6a57786370e in term 1.
I20260806 00:57:57.980916 25907 leader_election.cc:302] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75fdb5ac10a04820802de6a57786370e, 900b93eac37a4cb68d442d4ef7859989; no voters:
I20260806 00:57:57.981037 26198 raft_consensus.cc:2781] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:57:57.981202 26198 raft_consensus.cc:686] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e [term 1 LEADER]: Becoming Leader. State: Replica: 75fdb5ac10a04820802de6a57786370e, State: Running, Role: LEADER
I20260806 00:57:57.981307 26198 consensus_queue.cc:237] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } }
I20260806 00:57:57.981802 26237 tablet_replica.cc:439] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e: TxnStatusTablet state changed. Reason: New leader 75fdb5ac10a04820802de6a57786370e. Latest consensus state: current_term: 1 leader_uuid: "75fdb5ac10a04820802de6a57786370e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } } }
I20260806 00:57:57.981952 26237 tablet_replica.cc:442] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e: This TxnStatusTablet replica's current role is: LEADER
I20260806 00:57:57.982256 26239 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 00:57:57.982410 26239 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 00:57:57.982362 26197 consensus_queue.cc:1035] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:57:57.982331 25670 catalog_manager.cc:5495] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e reported cstate change: term changed from 0 to 1, leader changed from <none> to 75fdb5ac10a04820802de6a57786370e (127.23.242.2). New cstate: current_term: 1 leader_uuid: "75fdb5ac10a04820802de6a57786370e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } health_report { overall_health: UNKNOWN } } }
I20260806 00:57:57.984637 26197 consensus_queue.cc:1035] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:57:58.003895 26230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:58.001022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:36389 (local address 127.0.0.1:43170)
0806 00:57:58.001127 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:58.001141 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:58.001202 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:58.001474 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:58.001475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:58.001476 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 00:57:58.001571 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:58.001574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:58.002234 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:58.002235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:58.002944 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:58.002946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:58.003321 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:58.003322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:58.003388 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:58.003405 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 00:57:58.003687 (+ 282us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 00:57:58.003697 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 00:57:58.003766 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 00:57:58.003820 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 00:57:58.003906 26187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:58.001113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43170 (local address 127.23.242.2:36389)
0806 00:57:58.001163 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:58.001165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:58.001177 (+ 12us) server_negotiation.cc:407] Connection header received
0806 00:57:58.001326 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:58.001327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:58.001352 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:58.001414 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 00:57:58.001634 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:58.002141 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:58.003042 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:58.003242 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:58.003318 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:58.003513 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 00:57:58.003618 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 00:57:58.003701 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:58.003775 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 00:57:58.003798 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 00:57:58.006661 26119 raft_consensus.cc:1261] T 59fa09e81a6843438a18bcbdfad6b4c3 P 3e926b5bc3984f908ed53d24b091adb8 [term 1 FOLLOWER]: Refusing update from remote peer 75fdb5ac10a04820802de6a57786370e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:57:58.007056 25830 raft_consensus.cc:1261] T 59fa09e81a6843438a18bcbdfad6b4c3 P 900b93eac37a4cb68d442d4ef7859989 [term 1 FOLLOWER]: Refusing update from remote peer 75fdb5ac10a04820802de6a57786370e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:57:58.007169 26237 consensus_queue.cc:1035] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:57:58.007499 26198 consensus_queue.cc:1035] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e [LEADER]: Connected to new peer: Peer: permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:57:58.008127 26197 tablet_replica.cc:439] T 59fa09e81a6843438a18bcbdfad6b4c3 P 900b93eac37a4cb68d442d4ef7859989: TxnStatusTablet state changed. Reason: New leader 75fdb5ac10a04820802de6a57786370e. Latest consensus state: current_term: 1 leader_uuid: "75fdb5ac10a04820802de6a57786370e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } } }
I20260806 00:57:58.008227 26197 tablet_replica.cc:442] T 59fa09e81a6843438a18bcbdfad6b4c3 P 900b93eac37a4cb68d442d4ef7859989: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 00:57:58.008793 26246 tablet_replica.cc:439] T 59fa09e81a6843438a18bcbdfad6b4c3 P 3e926b5bc3984f908ed53d24b091adb8: TxnStatusTablet state changed. Reason: New leader 75fdb5ac10a04820802de6a57786370e. Latest consensus state: current_term: 1 leader_uuid: "75fdb5ac10a04820802de6a57786370e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } } }
I20260806 00:57:58.008915 26246 tablet_replica.cc:442] T 59fa09e81a6843438a18bcbdfad6b4c3 P 3e926b5bc3984f908ed53d24b091adb8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 00:57:58.009371 26198 tablet_replica.cc:439] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75fdb5ac10a04820802de6a57786370e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } } }
I20260806 00:57:58.009467 26198 tablet_replica.cc:442] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e: This TxnStatusTablet replica's current role is: LEADER
I20260806 00:57:58.009413 26237 tablet_replica.cc:439] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75fdb5ac10a04820802de6a57786370e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } } }
I20260806 00:57:58.009511 26237 tablet_replica.cc:442] T 59fa09e81a6843438a18bcbdfad6b4c3 P 75fdb5ac10a04820802de6a57786370e: This TxnStatusTablet replica's current role is: LEADER
I20260806 00:57:58.009656 26246 tablet_replica.cc:439] T 59fa09e81a6843438a18bcbdfad6b4c3 P 3e926b5bc3984f908ed53d24b091adb8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75fdb5ac10a04820802de6a57786370e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } } }
I20260806 00:57:58.009828 26246 tablet_replica.cc:442] T 59fa09e81a6843438a18bcbdfad6b4c3 P 3e926b5bc3984f908ed53d24b091adb8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 00:57:58.010210 26242 mvcc.cc:205] Tried to move back new op lower bound from 7315365388310683648 to 7315365388211884032. Current Snapshot: MvccSnapshot[applied={T|T < 7315365388310683648}]
I20260806 00:57:58.010424 26197 tablet_replica.cc:439] T 59fa09e81a6843438a18bcbdfad6b4c3 P 900b93eac37a4cb68d442d4ef7859989: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75fdb5ac10a04820802de6a57786370e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "900b93eac37a4cb68d442d4ef7859989" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46733 } } peers { permanent_uuid: "75fdb5ac10a04820802de6a57786370e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36389 } } peers { permanent_uuid: "3e926b5bc3984f908ed53d24b091adb8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 42861 } } }
I20260806 00:57:58.010525 26197 tablet_replica.cc:442] T 59fa09e81a6843438a18bcbdfad6b4c3 P 900b93eac37a4cb68d442d4ef7859989: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 00:57:58.017843 26235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:58.014049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37102 (local address 127.23.242.1:46733)
0806 00:57:58.014108 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:58.014111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:58.014122 (+ 11us) server_negotiation.cc:407] Connection header received
0806 00:57:58.014249 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:58.014251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:58.014272 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:58.014337 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 00:57:58.014520 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:58.015173 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:58.016671 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:58.016892 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:58.016970 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:58.017321 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 00:57:58.017519 (+ 198us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 00:57:58.017597 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:58.017748 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 00:57:58.017775 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 00:57:58.026522 26187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:58.021326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:54905 (local address 127.23.242.2:36389)
0806 00:57:58.021400 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:58.021402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:58.021558 (+ 156us) server_negotiation.cc:407] Connection header received
0806 00:57:58.021603 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:58.021604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:58.021614 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:58.021663 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 00:57:58.021996 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:58.023167 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:58.025295 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:58.025573 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:58.025877 (+ 304us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:58.026137 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 00:57:58.026173 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 00:57:58.026228 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:58.026417 (+ 189us) server_negotiation.cc:299] Negotiation successful
0806 00:57:58.026442 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 00:57:58.026532 26258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:58.021258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:36389 (local address 127.23.242.1:54905)
0806 00:57:58.021464 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:58.021476 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:58.021545 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:58.021684 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:58.021685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:58.021687 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 00:57:58.021750 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:58.021751 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:58.023482 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:58.023483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:58.024981 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:58.024985 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:58.025876 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:58.025878 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:58.025952 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:58.025982 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 00:57:58.026247 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 00:57:58.026250 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 00:57:58.026321 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 00:57:58.026379 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":117,"threads_started":1}
I20260806 00:57:58.041596 26235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:58.038472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:60805 (local address 127.23.242.1:46733)
0806 00:57:58.038612 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:58.038614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:58.038813 (+ 199us) server_negotiation.cc:407] Connection header received
0806 00:57:58.038873 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:58.038873 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:58.038883 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:58.038930 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 00:57:58.039109 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:58.039911 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:58.040848 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:58.041059 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:58.041148 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:58.041344 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 00:57:58.041379 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 00:57:58.041431 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:58.041487 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 00:57:58.041510 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260806 00:57:58.041648 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:58.038137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:46733 (local address 127.23.242.2:60805)
0806 00:57:58.038729 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:58.038742 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:58.038811 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:58.038941 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:58.038941 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:58.038943 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 00:57:58.039020 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:58.039022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:58.040001 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:58.040002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:58.040665 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:58.040676 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:58.041151 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:58.041152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:58.041219 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:58.041245 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 00:57:58.041439 (+ 194us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 00:57:58.041441 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 00:57:58.041486 (+ 45us) client_negotiation.cc:240] Negotiation successful
0806 00:57:58.041530 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":97,"threads_started":1}
I20260806 00:57:58.057296 25670 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:57:58.057433 25670 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:57:58.058249 25670 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7d05de3d237a44a2bedf639ef6dbd449: Sending DeleteTablet for 3 replicas of tablet a515a68831b1496496a390d2ba877a76
I20260806 00:57:58.058538 25810 tablet_service.cc:1514] Processing DeleteTablet for tablet a515a68831b1496496a390d2ba877a76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:57:58 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:55386
I20260806 00:57:58.058916 26098 tablet_service.cc:1514] Processing DeleteTablet for tablet a515a68831b1496496a390d2ba877a76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:57:58 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:41782
I20260806 00:57:58.059187 26265 tablet_replica.cc:330] T a515a68831b1496496a390d2ba877a76 P 3e926b5bc3984f908ed53d24b091adb8: stopping tablet replica
I20260806 00:57:58.059204 25955 tablet_service.cc:1514] Processing DeleteTablet for tablet a515a68831b1496496a390d2ba877a76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:57:58 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:32888
I20260806 00:57:58.059257 26265 raft_consensus.cc:2229] T a515a68831b1496496a390d2ba877a76 P 3e926b5bc3984f908ed53d24b091adb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:57:58.059301 26265 raft_consensus.cc:2258] T a515a68831b1496496a390d2ba877a76 P 3e926b5bc3984f908ed53d24b091adb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:57:58.059600 26265 ts_tablet_manager.cc:1905] T a515a68831b1496496a390d2ba877a76 P 3e926b5bc3984f908ed53d24b091adb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:57:58.060269 26266 tablet_replica.cc:330] T a515a68831b1496496a390d2ba877a76 P 75fdb5ac10a04820802de6a57786370e: stopping tablet replica
I20260806 00:57:58.060596 26266 raft_consensus.cc:2229] T a515a68831b1496496a390d2ba877a76 P 75fdb5ac10a04820802de6a57786370e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:57:58.060894 26266 raft_consensus.cc:2258] T a515a68831b1496496a390d2ba877a76 P 75fdb5ac10a04820802de6a57786370e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:57:58.061275 26266 ts_tablet_manager.cc:1905] T a515a68831b1496496a390d2ba877a76 P 75fdb5ac10a04820802de6a57786370e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:57:58.062034 26265 ts_tablet_manager.cc:1918] T a515a68831b1496496a390d2ba877a76 P 3e926b5bc3984f908ed53d24b091adb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 00:57:58.062105 26265 log.cc:1196] T a515a68831b1496496a390d2ba877a76 P 3e926b5bc3984f908ed53d24b091adb8: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-2/wal/wals/a515a68831b1496496a390d2ba877a76
I20260806 00:57:58.062350 26265 ts_tablet_manager.cc:1939] T a515a68831b1496496a390d2ba877a76 P 3e926b5bc3984f908ed53d24b091adb8: Deleting consensus metadata
I20260806 00:57:58.062772 25654 catalog_manager.cc:4841] TS 3e926b5bc3984f908ed53d24b091adb8 (127.23.242.3:42861): tablet a515a68831b1496496a390d2ba877a76 (table test-table [id=2baf9dba5dbc4be0b997ac674666667c]) successfully deleted
I20260806 00:57:58.064051 26266 ts_tablet_manager.cc:1918] T a515a68831b1496496a390d2ba877a76 P 75fdb5ac10a04820802de6a57786370e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 00:57:58.064152 26266 log.cc:1196] T a515a68831b1496496a390d2ba877a76 P 75fdb5ac10a04820802de6a57786370e: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-1/wal/wals/a515a68831b1496496a390d2ba877a76
I20260806 00:57:58.064420 26266 ts_tablet_manager.cc:1939] T a515a68831b1496496a390d2ba877a76 P 75fdb5ac10a04820802de6a57786370e: Deleting consensus metadata
I20260806 00:57:58.064824 25654 catalog_manager.cc:4841] TS 75fdb5ac10a04820802de6a57786370e (127.23.242.2:36389): tablet a515a68831b1496496a390d2ba877a76 (table test-table [id=2baf9dba5dbc4be0b997ac674666667c]) successfully deleted
I20260806 00:57:58.066860 26264 tablet_replica.cc:330] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989: stopping tablet replica
I20260806 00:57:58.067023 26264 raft_consensus.cc:2229] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989 [term 1 LEADER]: Raft consensus shutting down.
I20260806 00:57:58.067165 26264 raft_consensus.cc:2258] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:57:58.067847 26264 ts_tablet_manager.cc:1905] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:57:58.071488 26264 ts_tablet_manager.cc:1918] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 00:57:58.071662 26264 log.cc:1196] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785977869987768-24520-0/minicluster-data/ts-0/wal/wals/a515a68831b1496496a390d2ba877a76
I20260806 00:57:58.072341 26264 ts_tablet_manager.cc:1939] T a515a68831b1496496a390d2ba877a76 P 900b93eac37a4cb68d442d4ef7859989: Deleting consensus metadata
Aug 06 00:57:58 dist-test-slave-dhph krb5kdc[25621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977877, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.242.1@KRBTEST.COM
I20260806 00:57:58.073254 25656 catalog_manager.cc:4841] TS 900b93eac37a4cb68d442d4ef7859989 (127.23.242.1:46733): tablet a515a68831b1496496a390d2ba877a76 (table test-table [id=2baf9dba5dbc4be0b997ac674666667c]) successfully deleted
I20260806 00:57:58.078052 26235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:58.060607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37118 (local address 127.23.242.1:46733)
0806 00:57:58.060657 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:58.060659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:58.063779 (+ 3120us) server_negotiation.cc:407] Connection header received
0806 00:57:58.063852 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:58.063854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:58.063868 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:58.063935 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:58.066256 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:58.067132 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:58.068461 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:58.068776 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:58.068828 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:58.073897 (+ 5069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:58.073907 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:58.073908 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:58.073917 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:58.076353 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:58.076888 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:58.076892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:58.076892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:58.076979 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:58.077201 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:58.077202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:58.077202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:58.077400 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:58.077545 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:58.077814 (+ 269us) server_negotiation.cc:299] Negotiation successful
0806 00:57:58.077924 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":28}
W20260806 00:57:58.079071 25871 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37118
I20260806 00:57:58.091466 25721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:58.082988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48326 (local address 127.23.242.62:39189)
0806 00:57:58.083050 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:58.083052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:58.083147 (+ 95us) server_negotiation.cc:407] Connection header received
0806 00:57:58.083201 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:58.083202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:58.083216 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:58.083280 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:58.083983 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:58.084785 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:58.085771 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:58.086088 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:58.086162 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:58.087834 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:58.087835 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:58.087836 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:58.087844 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:58.089637 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:58.090323 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:58.090324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:58.090325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:58.090392 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:58.090691 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:58.090693 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:58.090693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:58.090945 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:58.091068 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:58.091272 (+ 204us) server_negotiation.cc:299] Negotiation successful
0806 00:57:58.091361 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260806 00:57:58.092181 25670 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48326
I20260806 00:57:58.092986 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 25743
I20260806 00:57:58.102280 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 25887
I20260806 00:57:58.110029 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 26030
I20260806 00:57:58.117955 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 25636
2026-08-06T00:57:58Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3212 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 00:57:58 dist-test-slave-dhph krb5kdc[26273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 00:57:58 dist-test-slave-dhph krb5kdc[26273](info): set up 2 sockets
Aug 06 00:57:58 dist-test-slave-dhph krb5kdc[26273](info): commencing operation
krb5kdc: starting...
W20260806 00:57:59.296087 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 00:57:59 dist-test-slave-dhph krb5kdc[26273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T00:57:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T00:57:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 00:57:59.456276 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:41397
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:41397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:57:59.547242 26286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:57:59.547479 26286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:57:59.547508 26286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:57:59.548830 26286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 00:57:59.548883 26286 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 00:57:59.548898 26286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:57:59.548911 26286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:57:59.548924 26286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 00:57:59.548938 26286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 00:57:59.550630 26286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:41397
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:41397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:57:59.550920 26286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:57:59.551201 26286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:57:59.556864 26294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:57:59.556949 26291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:57:59.556937 26292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:57:59.556941 26286 server_base.cc:1034] running on GCE node
I20260806 00:57:59.557476 26286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:57:59.557775 26286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:57:59.559000 26286 hybrid_clock.cc:648] HybridClock initialized: now 1785977879558990 us; error 51 us; skew 500 ppm
Aug 06 00:57:59 dist-test-slave-dhph krb5kdc[26273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977879, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:57:59.561985 26286 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 00:57:59.562566 26286 webserver.cc:460] Webserver started at http://127.23.242.62:33143/ using document root <none> and password file <none>
I20260806 00:57:59.562939 26286 fs_manager.cc:359] Metadata directory not provided
I20260806 00:57:59.563015 26286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:57:59.563196 26286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:57:59.564328 26286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "70823b4c6d02429ca702661edb1041e8"
format_stamp: "Formatted at 2026-08-06 00:57:59 on dist-test-slave-dhph"
server_key: "66fc5ffb143162c3329ee154287d03b6"
server_key_iv: "da002ac25e78c3537736d2fd27837857"
server_key_version: "encryptionkey@0"
I20260806 00:57:59.564720 26286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "70823b4c6d02429ca702661edb1041e8"
format_stamp: "Formatted at 2026-08-06 00:57:59 on dist-test-slave-dhph"
server_key: "66fc5ffb143162c3329ee154287d03b6"
server_key_iv: "da002ac25e78c3537736d2fd27837857"
server_key_version: "encryptionkey@0"
I20260806 00:57:59.566200 26286 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 00:57:59.567363 26301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:59.567762 26286 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:57:59.568003 26286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "70823b4c6d02429ca702661edb1041e8"
format_stamp: "Formatted at 2026-08-06 00:57:59 on dist-test-slave-dhph"
server_key: "66fc5ffb143162c3329ee154287d03b6"
server_key_iv: "da002ac25e78c3537736d2fd27837857"
server_key_version: "encryptionkey@0"
I20260806 00:57:59.568149 26286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:57:59.582218 26286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:57:59.584841 26286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:57:59.585024 26286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:57:59.590747 26286 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:41397
I20260806 00:57:59.590803 26364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:41397 every 8 connection(s)
I20260806 00:57:59.591156 26286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 00:57:59.591813 26365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:57:59.593330 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 26286
I20260806 00:57:59.593417 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 00:57:59.593607 24520 external_mini_cluster.cc:1428] Setting key 4cd675d13e1b48e918b4cb7e0257299c
I20260806 00:57:59.594329 26365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70823b4c6d02429ca702661edb1041e8: Bootstrap starting.
I20260806 00:57:59.595005 26365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70823b4c6d02429ca702661edb1041e8: Neither blocks nor log segments found. Creating new log.
I20260806 00:57:59.595398 26365 log.cc:824] T 00000000000000000000000000000000 P 70823b4c6d02429ca702661edb1041e8: Log is configured to *not* fsync() on all Append() calls
I20260806 00:57:59.596199 26365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70823b4c6d02429ca702661edb1041e8: No bootstrap required, opened a new log
I20260806 00:57:59.597911 26365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70823b4c6d02429ca702661edb1041e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70823b4c6d02429ca702661edb1041e8" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 41397 } }
I20260806 00:57:59.598121 26365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70823b4c6d02429ca702661edb1041e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:57:59.598163 26365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70823b4c6d02429ca702661edb1041e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70823b4c6d02429ca702661edb1041e8, State: Initialized, Role: FOLLOWER
I20260806 00:57:59.598326 26365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70823b4c6d02429ca702661edb1041e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70823b4c6d02429ca702661edb1041e8" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 41397 } }
I20260806 00:57:59.598407 26365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70823b4c6d02429ca702661edb1041e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 00:57:59.598436 26365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70823b4c6d02429ca702661edb1041e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 00:57:59.598474 26365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70823b4c6d02429ca702661edb1041e8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:57:59.599365 26365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70823b4c6d02429ca702661edb1041e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70823b4c6d02429ca702661edb1041e8" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 41397 } }
I20260806 00:57:59.599463 26365 leader_election.cc:302] T 00000000000000000000000000000000 P 70823b4c6d02429ca702661edb1041e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70823b4c6d02429ca702661edb1041e8; no voters:
I20260806 00:57:59.599674 26365 leader_election.cc:288] T 00000000000000000000000000000000 P 70823b4c6d02429ca702661edb1041e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 00:57:59.599754 26370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70823b4c6d02429ca702661edb1041e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:57:59.599877 26370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70823b4c6d02429ca702661edb1041e8 [term 1 LEADER]: Becoming Leader. State: Replica: 70823b4c6d02429ca702661edb1041e8, State: Running, Role: LEADER
I20260806 00:57:59.599961 26370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70823b4c6d02429ca702661edb1041e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70823b4c6d02429ca702661edb1041e8" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 41397 } }
I20260806 00:57:59.599968 26365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70823b4c6d02429ca702661edb1041e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 00:57:59.600283 26370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70823b4c6d02429ca702661edb1041e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70823b4c6d02429ca702661edb1041e8. Latest consensus state: current_term: 1 leader_uuid: "70823b4c6d02429ca702661edb1041e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70823b4c6d02429ca702661edb1041e8" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 41397 } } }
I20260806 00:57:59.600294 26371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70823b4c6d02429ca702661edb1041e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70823b4c6d02429ca702661edb1041e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70823b4c6d02429ca702661edb1041e8" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 41397 } } }
I20260806 00:57:59.600369 26370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70823b4c6d02429ca702661edb1041e8 [sys.catalog]: This master's current role is: LEADER
I20260806 00:57:59.600394 26371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70823b4c6d02429ca702661edb1041e8 [sys.catalog]: This master's current role is: LEADER
I20260806 00:57:59.601255 26374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 00:57:59.601910 26374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 06 00:57:59 dist-test-slave-dhph krb5kdc[26273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:57:59.603685 26374 catalog_manager.cc:1347] Generated new cluster ID: cffa248ade8e4e0ba4e3d76d51de692b
I20260806 00:57:59.603746 26374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 00:57:59.608289 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:59.594199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48804 (local address 127.23.242.62:41397)
0806 00:57:59.594411 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:59.594420 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:59.594451 (+ 31us) server_negotiation.cc:407] Connection header received
0806 00:57:59.595012 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:59.595020 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:59.595071 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:59.595196 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:59.596040 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:59.597609 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:59.598994 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:59.599503 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:59.599634 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:59.603406 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:59.603412 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:59.603413 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:59.603428 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:59.605950 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:59.606404 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:59.606409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:59.606409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:59.606545 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:59.607152 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:59.607153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:59.607153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:59.607538 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:59.607756 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:59.607848 (+ 92us) server_negotiation.cc:299] Negotiation successful
0806 00:57:59.608027 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":122,"threads_started":1}
I20260806 00:57:59.616878 26374 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 00:57:59.617664 26374 catalog_manager.cc:1504] Loading token signing keys...
I20260806 00:57:59.628253 26374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70823b4c6d02429ca702661edb1041e8: Generated new TSK 0
I20260806 00:57:59.628477 26374 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.242.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
I20260806 00:57:59.662402 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:0
--local_ip_for_outbound_sockets=127.23.242.1
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:41397
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:57:59.755244 26392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:57:59.755460 26392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:57:59.755491 26392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:57:59.756739 26392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:57:59.756796 26392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:57:59.756819 26392 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 00:57:59.756832 26392 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 00:57:59.756850 26392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 00:57:59.758539 26392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.242.62:41397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:57:59.758806 26392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:57:59.758997 26392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:57:59.764771 26400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:57:59.764743 26397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:57:59.764778 26398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:57:59.765084 26392 server_base.cc:1034] running on GCE node
I20260806 00:57:59.765326 26392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:57:59.765628 26392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:57:59.766789 26392 hybrid_clock.cc:648] HybridClock initialized: now 1785977879766763 us; error 39 us; skew 500 ppm
Aug 06 00:57:59 dist-test-slave-dhph krb5kdc[26273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977879, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:57:59.769995 26392 init.cc:377] Logged in from keytab as kudu/127.23.242.1@KRBTEST.COM (short username kudu)
I20260806 00:57:59.770519 26392 webserver.cc:460] Webserver started at http://127.23.242.1:33985/ using document root <none> and password file <none>
I20260806 00:57:59.770838 26392 fs_manager.cc:359] Metadata directory not provided
I20260806 00:57:59.770895 26392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:57:59.771047 26392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:57:59.772317 26392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-0/data/instance:
uuid: "e57237ca962b4e81a69d191e8d9c1ab4"
format_stamp: "Formatted at 2026-08-06 00:57:59 on dist-test-slave-dhph"
server_key: "23120a3198b01c49698e96f6be599afb"
server_key_iv: "0e44b96120dd306aab99725c7624dd4c"
server_key_version: "encryptionkey@0"
I20260806 00:57:59.772773 26392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance:
uuid: "e57237ca962b4e81a69d191e8d9c1ab4"
format_stamp: "Formatted at 2026-08-06 00:57:59 on dist-test-slave-dhph"
server_key: "23120a3198b01c49698e96f6be599afb"
server_key_iv: "0e44b96120dd306aab99725c7624dd4c"
server_key_version: "encryptionkey@0"
I20260806 00:57:59.774286 26392 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 00:57:59.775319 26407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:59.775817 26392 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:57:59.775921 26392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "e57237ca962b4e81a69d191e8d9c1ab4"
format_stamp: "Formatted at 2026-08-06 00:57:59 on dist-test-slave-dhph"
server_key: "23120a3198b01c49698e96f6be599afb"
server_key_iv: "0e44b96120dd306aab99725c7624dd4c"
server_key_version: "encryptionkey@0"
I20260806 00:57:59.775977 26392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:57:59.793746 26392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:57:59.796324 26392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:57:59.796465 26392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:57:59.797008 26392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:57:59.797106 26392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:59.797150 26392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:57:59.797191 26392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 00:57:59 dist-test-slave-dhph krb5kdc[26273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977879, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:57:59.805543 26392 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:36631
I20260806 00:57:59.805631 26524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:36631 every 8 connection(s)
I20260806 00:57:59.806444 26392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 00:57:59.808764 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 26392
I20260806 00:57:59.808861 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance
I20260806 00:57:59.809087 24520 external_mini_cluster.cc:1428] Setting key 0938201bb29a366343a4bcdc9473b0d1
I20260806 00:57:59.809628 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:59.797626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:51373 (local address 127.23.242.62:41397)
0806 00:57:59.797676 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:59.797678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:59.797904 (+ 226us) server_negotiation.cc:407] Connection header received
0806 00:57:59.798519 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:59.798520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:59.798534 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:59.798615 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:59.799465 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:59.800303 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:59.802306 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:59.802668 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:59.802753 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:59.805931 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:59.805933 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:59.805933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:59.805942 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:59.807816 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:59.808326 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:59.808327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:59.808327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:59.808400 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:59.808713 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:59.808714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:59.808715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:59.808994 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:59.809108 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:59.809423 (+ 315us) server_negotiation.cc:299] Negotiation successful
0806 00:57:59.809524 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 00:57:59.809862 26416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:59.797609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:41397 (local address 127.23.242.1:51373)
0806 00:57:59.797782 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:59.797818 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:59.798386 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:59.798681 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:59.798685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:59.799045 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:57:59.799365 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:59.799372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:59.800418 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:59.800420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:59.802202 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:59.802207 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:59.802806 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:59.802808 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:59.802991 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:59.803581 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:57:59.803594 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:57:59.805793 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:57:59.807948 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:59.807957 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:59.807959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:59.808224 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:59.808475 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:59.808477 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:59.808477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:59.808622 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:59.809098 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:57:59.809106 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:57:59.809279 (+ 173us) client_negotiation.cc:769] Sending connection context
0806 00:57:59.809401 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 00:57:59.809577 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
I20260806 00:57:59.810669 26525 heartbeater.cc:347] Connected to a master server at 127.23.242.62:41397
I20260806 00:57:59.810779 26525 heartbeater.cc:464] Registering TS with master...
I20260806 00:57:59.811124 26525 heartbeater.cc:511] Master 127.23.242.62:41397 requested a full tablet report, sending...
I20260806 00:57:59.812196 26318 ts_manager.cc:194] Registered new tserver with Master: e57237ca962b4e81a69d191e8d9c1ab4 (127.23.242.1:36631)
I20260806 00:57:59.813030 26318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.1@KRBTEST.COM'} at 127.23.242.1:51373
WARNING: no policy specified for kudu/127.23.242.2@KRBTEST.COM; defaulting to no policy
I20260806 00:57:59.821457 26531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:59.812078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:41397 (local address 127.23.242.1:54785)
0806 00:57:59.812275 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:59.812287 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:59.812362 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:59.812494 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:59.812495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:59.812724 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:57:59.812811 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:59.812813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:59.814175 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:59.814177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:59.814913 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:59.814916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:59.815721 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:59.815723 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:59.815830 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:59.816445 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:57:59.816451 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:57:59.816975 (+ 524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:57:59.819636 (+ 2661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:59.819638 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:59.819639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:59.819875 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:59.820432 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:59.820433 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:59.820433 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:59.820525 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:59.821095 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:57:59.821095 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:57:59.821165 (+ 70us) client_negotiation.cc:769] Sending connection context
0806 00:57:59.821255 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 00:57:59.821367 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":127,"threads_started":1}
I20260806 00:57:59.821686 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:59.812125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:54785 (local address 127.23.242.62:41397)
0806 00:57:59.812174 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:59.812175 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:59.812337 (+ 162us) server_negotiation.cc:407] Connection header received
0806 00:57:59.812424 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:59.812425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:59.812435 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:59.812488 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:59.812906 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:59.813968 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:59.815014 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:59.815575 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:59.815709 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:59.817161 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:59.817162 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:59.817163 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:59.817170 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:59.819524 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:59.820223 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:59.820225 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:59.820225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:59.820310 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:59.820687 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:59.820689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:59.820689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:59.820918 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:59.821075 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:59.821370 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 00:57:59.821469 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
I20260806 00:57:59.849174 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:0
--local_ip_for_outbound_sockets=127.23.242.2
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:41397
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:57:59.944896 26534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:57:59.945166 26534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:57:59.945199 26534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:57:59.946614 26534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:57:59.946667 26534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:57:59.946692 26534 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 00:57:59.946707 26534 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 00:57:59.946770 26534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 00:57:59.948508 26534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.242.62:41397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:57:59.948834 26534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:57:59.949126 26534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:57:59.955518 26539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:57:59.955803 26542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:57:59.955825 26540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:57:59.955833 26534 server_base.cc:1034] running on GCE node
I20260806 00:57:59.956069 26534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:57:59.956357 26534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:57:59.957649 26534 hybrid_clock.cc:648] HybridClock initialized: now 1785977879957633 us; error 33 us; skew 500 ppm
Aug 06 00:57:59 dist-test-slave-dhph krb5kdc[26273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977879, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:57:59.960762 26534 init.cc:377] Logged in from keytab as kudu/127.23.242.2@KRBTEST.COM (short username kudu)
I20260806 00:57:59.961375 26534 webserver.cc:460] Webserver started at http://127.23.242.2:33365/ using document root <none> and password file <none>
I20260806 00:57:59.961673 26534 fs_manager.cc:359] Metadata directory not provided
I20260806 00:57:59.961755 26534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:57:59.961934 26534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:57:59.963233 26534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-1/data/instance:
uuid: "4650ad0a2c9f4d308af8035b8b5efeb7"
format_stamp: "Formatted at 2026-08-06 00:57:59 on dist-test-slave-dhph"
server_key: "c913c899c7bf2ffa1a3bf3115032e3c4"
server_key_iv: "3fa561da8832b259be040a9bd036f00d"
server_key_version: "encryptionkey@0"
I20260806 00:57:59.963670 26534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance:
uuid: "4650ad0a2c9f4d308af8035b8b5efeb7"
format_stamp: "Formatted at 2026-08-06 00:57:59 on dist-test-slave-dhph"
server_key: "c913c899c7bf2ffa1a3bf3115032e3c4"
server_key_iv: "3fa561da8832b259be040a9bd036f00d"
server_key_version: "encryptionkey@0"
I20260806 00:57:59.965401 26534 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 00:57:59.966430 26549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:59.967200 26534 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:57:59.967371 26534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "4650ad0a2c9f4d308af8035b8b5efeb7"
format_stamp: "Formatted at 2026-08-06 00:57:59 on dist-test-slave-dhph"
server_key: "c913c899c7bf2ffa1a3bf3115032e3c4"
server_key_iv: "3fa561da8832b259be040a9bd036f00d"
server_key_version: "encryptionkey@0"
I20260806 00:57:59.967475 26534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:57:59.981418 26534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:57:59.984474 26534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:57:59.984694 26534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:57:59.985430 26534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:57:59.985487 26534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:57:59.985533 26534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:57:59.985549 26534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 00:57:59 dist-test-slave-dhph krb5kdc[26273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977879, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:57:59.994762 26534 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:33789
I20260806 00:57:59.994918 26665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:33789 every 8 connection(s)
I20260806 00:57:59.995236 26534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 00:57:59.996731 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 26534
I20260806 00:57:59.996851 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance
I20260806 00:57:59.997038 24520 external_mini_cluster.cc:1428] Setting key e339e2b3ed9505d03011d93b7a18c9ee
I20260806 00:57:59.997973 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:59.985930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:53811 (local address 127.23.242.62:41397)
0806 00:57:59.986006 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0806 00:57:59.986010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 00:57:59.986211 (+ 201us) server_negotiation.cc:407] Connection header received
0806 00:57:59.986864 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:57:59.986865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:57:59.986883 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:57:59.986972 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:57:59.987829 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:59.988766 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:59.990158 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:57:59.990548 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:57:59.990673 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:59.994252 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:57:59.994254 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:57:59.994255 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:57:59.994263 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:57:59.996130 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:59.996614 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:59.996616 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:59.996617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:59.996701 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:57:59.997088 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:57:59.997089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:57:59.997089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:57:59.997265 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:57:59.997339 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 00:57:59.997787 (+ 448us) server_negotiation.cc:299] Negotiation successful
0806 00:57:59.997879 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 00:57:59.998222 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:57:59.985899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:41397 (local address 127.23.242.2:53811)
0806 00:57:59.986063 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 00:57:59.986105 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 00:57:59.986754 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:57:59.987038 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:57:59.987041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:57:59.987397 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:57:59.987634 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:59.987640 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:59.988921 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:59.988922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:59.990036 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:57:59.990039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:57:59.990693 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:57:59.990695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:57:59.990851 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:57:59.991594 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:57:59.991622 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:57:59.994122 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:57:59.996242 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:59.996249 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:59.996251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:59.996513 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:59.996869 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:57:59.996870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:57:59.996871 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:57:59.997009 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:57:59.997431 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:57:59.997440 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:57:59.997607 (+ 167us) client_negotiation.cc:769] Sending connection context
0806 00:57:59.997736 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 00:57:59.997940 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260806 00:57:59.998821 26666 heartbeater.cc:347] Connected to a master server at 127.23.242.62:41397
I20260806 00:57:59.998924 26666 heartbeater.cc:464] Registering TS with master...
I20260806 00:57:59.999133 26666 heartbeater.cc:511] Master 127.23.242.62:41397 requested a full tablet report, sending...
I20260806 00:57:59.999552 26317 ts_manager.cc:194] Registered new tserver with Master: 4650ad0a2c9f4d308af8035b8b5efeb7 (127.23.242.2:33789)
I20260806 00:58:00.000088 26317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.2@KRBTEST.COM'} at 127.23.242.2:53811
WARNING: no policy specified for kudu/127.23.242.3@KRBTEST.COM; defaulting to no policy
I20260806 00:58:00.008827 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.000505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:59405 (local address 127.23.242.62:41397)
0806 00:58:00.000548 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:00.000550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:00.000941 (+ 391us) server_negotiation.cc:407] Connection header received
0806 00:58:00.001003 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:00.001004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:00.001019 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:00.001096 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:00.001802 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.002461 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.003853 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.004075 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.004146 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.005603 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:00.005605 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:00.005605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:00.005614 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:00.007409 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:00.007883 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:00.007888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:00.007889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:00.007948 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:00.008191 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:00.008192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:00.008192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:00.008352 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:00.008406 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:00.008667 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 00:58:00.008758 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 00:58:00.008889 26673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.000588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:41397 (local address 127.23.242.2:59405)
0806 00:58:00.000850 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:00.000862 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:00.000937 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:00.001160 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:00.001162 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:00.001444 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:00.001584 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.001587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.002732 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.002734 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.003641 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.003646 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.004165 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.004167 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.004262 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.004877 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:00.004883 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:00.005421 (+ 538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:00.007508 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:00.007509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:00.007510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:00.007782 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:00.008032 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:00.008033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:00.008033 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:00.008118 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:00.008422 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:00.008423 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:00.008495 (+ 72us) client_negotiation.cc:769] Sending connection context
0806 00:58:00.008600 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 00:58:00.008745 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":184,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
I20260806 00:58:00.042781 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:0
--local_ip_for_outbound_sockets=127.23.242.3
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:41397
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:37395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:00.139017 26675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:00.139272 26675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:00.139302 26675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:00.140594 26675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:00.140647 26675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:00.140671 26675 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 00:58:00.140682 26675 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 00:58:00.140708 26675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 00:58:00.142222 26675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:37395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.242.62:41397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:00.142446 26675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:00.142727 26675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 00:58:00.149216 26675 server_base.cc:1034] running on GCE node
W20260806 00:58:00.149024 26680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:00.149024 26681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:00.149024 26683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:00.149739 26675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:00.150041 26675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:00.151262 26675 hybrid_clock.cc:648] HybridClock initialized: now 1785977880151267 us; error 84 us; skew 500 ppm
Aug 06 00:58:00 dist-test-slave-dhph krb5kdc[26273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977880, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:00.155005 26675 init.cc:377] Logged in from keytab as kudu/127.23.242.3@KRBTEST.COM (short username kudu)
I20260806 00:58:00.155632 26675 webserver.cc:460] Webserver started at http://127.23.242.3:45127/ using document root <none> and password file <none>
I20260806 00:58:00.155952 26675 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:00.156009 26675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:00.156152 26675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:00.157290 26675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-2/data/instance:
uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc"
format_stamp: "Formatted at 2026-08-06 00:58:00 on dist-test-slave-dhph"
server_key: "0d149f0c7c1517e68fe8b38df2881a10"
server_key_iv: "7b893b79eda9b957f74c5597d6d67d7e"
server_key_version: "encryptionkey@0"
I20260806 00:58:00.157676 26675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance:
uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc"
format_stamp: "Formatted at 2026-08-06 00:58:00 on dist-test-slave-dhph"
server_key: "0d149f0c7c1517e68fe8b38df2881a10"
server_key_iv: "7b893b79eda9b957f74c5597d6d67d7e"
server_key_version: "encryptionkey@0"
I20260806 00:58:00.159278 26675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 00:58:00.160475 26691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:00.160755 26675 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:00.161011 26675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc"
format_stamp: "Formatted at 2026-08-06 00:58:00 on dist-test-slave-dhph"
server_key: "0d149f0c7c1517e68fe8b38df2881a10"
server_key_iv: "7b893b79eda9b957f74c5597d6d67d7e"
server_key_version: "encryptionkey@0"
I20260806 00:58:00.161160 26675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:00.168452 26675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:00.172123 26675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:00.172299 26675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:00.173024 26675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:00.173082 26675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:00.173120 26675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:00.173139 26675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 00:58:00 dist-test-slave-dhph krb5kdc[26273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977880, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:00.182039 26675 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:39607
I20260806 00:58:00.182076 26811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:39607 every 8 connection(s)
I20260806 00:58:00.182439 26675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
I20260806 00:58:00.184846 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.173536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:49587 (local address 127.23.242.62:41397)
0806 00:58:00.173586 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:00.173588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:00.173924 (+ 336us) server_negotiation.cc:407] Connection header received
0806 00:58:00.174483 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:00.174484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:00.174498 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:00.174548 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:00.175238 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.175825 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.177598 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.177938 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.178225 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.181421 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:00.181423 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:00.181424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:00.181433 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:00.183346 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:00.183824 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:00.183825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:00.183825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:00.183879 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:00.184151 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:00.184152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:00.184152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:00.184299 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:00.184353 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:00.184691 (+ 338us) server_negotiation.cc:299] Negotiation successful
0806 00:58:00.184778 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 00:58:00.185048 26701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.173556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:41397 (local address 127.23.242.3:49587)
0806 00:58:00.173792 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:00.173837 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:00.174360 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:00.174660 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:00.174664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:00.174927 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:00.175121 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.175125 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.175934 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.175935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.177151 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.177161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.178238 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.178240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.178472 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.179183 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:00.179195 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:00.181295 (+ 2100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:00.183480 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:00.183487 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:00.183489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:00.183747 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:00.183962 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:00.183963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:00.183963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:00.184081 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:00.184400 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:00.184408 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:00.184546 (+ 138us) client_negotiation.cc:769] Sending connection context
0806 00:58:00.184657 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 00:58:00.184805 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":98,"threads_started":1}
I20260806 00:58:00.185587 26812 heartbeater.cc:347] Connected to a master server at 127.23.242.62:41397
I20260806 00:58:00.185689 26812 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:00.185933 26812 heartbeater.cc:511] Master 127.23.242.62:41397 requested a full tablet report, sending...
I20260806 00:58:00.186436 26318 ts_manager.cc:194] Registered new tserver with Master: 6e1948e9fd0b4766b3d8ac356d9d9fbc (127.23.242.3:39607)
I20260806 00:58:00.187047 26318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.3@KRBTEST.COM'} at 127.23.242.3:49587
I20260806 00:58:00.190976 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 26675
I20260806 00:58:00.191094 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance
I20260806 00:58:00.191311 24520 external_mini_cluster.cc:1428] Setting key 273eb526563f3dcca5c299a7d8a2303a
I20260806 00:58:00.192647 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 00:58:00.194839 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.187112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:47115 (local address 127.23.242.62:41397)
0806 00:58:00.187169 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:00.187172 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:00.187301 (+ 129us) server_negotiation.cc:407] Connection header received
0806 00:58:00.187348 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:00.187349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:00.187361 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:00.187427 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:00.187871 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.188735 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.189618 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.189851 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.189961 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.191298 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:00.191299 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:00.191300 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:00.191310 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:00.193391 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:00.193874 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:00.193875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:00.193876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:00.193935 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:00.194216 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:00.194217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:00.194217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:00.194353 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:00.194417 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:00.194689 (+ 272us) server_negotiation.cc:299] Negotiation successful
0806 00:58:00.194771 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 00:58:00.194950 26818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.186880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:41397 (local address 127.23.242.3:47115)
0806 00:58:00.187216 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:00.187227 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:00.187289 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:00.187446 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:00.187447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:00.187682 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:00.187761 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.187764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.188869 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.188870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.189458 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.189460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.189995 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.189996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.190072 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.190721 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:00.190727 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:00.191165 (+ 438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:00.193488 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:00.193490 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:00.193491 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:00.193768 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:00.194016 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:00.194017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:00.194018 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:00.194109 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:00.194433 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:00.194434 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:00.194494 (+ 60us) client_negotiation.cc:769] Sending connection context
0806 00:58:00.194610 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 00:58:00.194759 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":87,"threads_started":1}
Aug 06 00:58:00 dist-test-slave-dhph krb5kdc[26273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 00:58:00 dist-test-slave-dhph krb5kdc[26273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:00.222033 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.211856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48812 (local address 127.23.242.62:41397)
0806 00:58:00.211941 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:00.211943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:00.211955 (+ 12us) server_negotiation.cc:407] Connection header received
0806 00:58:00.211991 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:00.211992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:00.212001 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:00.212052 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:00.212937 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.213442 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.214555 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.214953 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.215024 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.218221 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:00.218223 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:00.218224 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:00.218232 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:00.220280 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:00.220927 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:00.220929 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:00.220929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:00.221012 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:00.221422 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:00.221423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:00.221423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:00.221574 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:00.221685 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:00.221866 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 00:58:00.221954 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 00:58:00.224327 26318 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 00:58:00.226178 26318 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 00:58:00.233525 26829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.228963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:33789 (local address 127.0.0.1:45398)
0806 00:58:00.229334 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:00.229381 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:00.229470 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:00.229799 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:00.229803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:00.229806 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:00.229883 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.229886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.231229 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.231231 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.232120 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.232122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.233109 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.233110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.233241 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.233257 (+ 16us) client_negotiation.cc:769] Sending connection context
0806 00:58:00.233341 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 00:58:00.233409 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":66,"threads_started":1}
I20260806 00:58:00.233537 26831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.229100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45398 (local address 127.23.242.2:33789)
0806 00:58:00.229308 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:00.229313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:00.229449 (+ 136us) server_negotiation.cc:407] Connection header received
0806 00:58:00.229615 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:00.229618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:00.229637 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:00.229720 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:00.229964 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.231125 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.232243 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.232997 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.233088 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.233148 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:00.233372 (+ 224us) server_negotiation.cc:299] Negotiation successful
0806 00:58:00.233422 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":117,"threads_started":1}
I20260806 00:58:00.233804 26830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.228963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:39607 (local address 127.0.0.1:39620)
0806 00:58:00.229444 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:00.229455 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:00.229508 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:00.229839 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:00.229840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:00.229841 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:00.229883 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.229885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.231274 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.231275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.232575 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.232578 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.233541 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.233542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.233620 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.233624 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 00:58:00.233681 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 00:58:00.233735 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":28,"thread_start_us":98,"threads_started":1}
I20260806 00:58:00.234062 26833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.229259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39620 (local address 127.23.242.3:39607)
0806 00:58:00.229681 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:00.229685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:00.229694 (+ 9us) server_negotiation.cc:407] Connection header received
0806 00:58:00.229732 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:00.229734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:00.229748 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:00.229957 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:00.230009 (+ 52us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.231161 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.232674 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.233380 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.233447 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.233501 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:00.233905 (+ 404us) server_negotiation.cc:299] Negotiation successful
0806 00:58:00.233950 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":95,"threads_started":1}
I20260806 00:58:00.234787 26599 tablet_service.cc:1467] Processing CreateTablet for tablet 2dfcce24f52c4309b13154a54335376d (DEFAULT_TABLE table=test-table [id=87a79ddf4b384256b19120ad2616c765]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:00.234774 26744 tablet_service.cc:1467] Processing CreateTablet for tablet 2dfcce24f52c4309b13154a54335376d (DEFAULT_TABLE table=test-table [id=87a79ddf4b384256b19120ad2616c765]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:00.235162 26599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dfcce24f52c4309b13154a54335376d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:00.235167 26744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dfcce24f52c4309b13154a54335376d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:00.238302 26835 tablet_bootstrap.cc:492] T 2dfcce24f52c4309b13154a54335376d P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Bootstrap starting.
I20260806 00:58:00.238708 26834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.228963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:36631 (local address 127.0.0.1:34164)
0806 00:58:00.229635 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:00.229647 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:00.229696 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:00.230004 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:00.230005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:00.230007 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:00.230065 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.230067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.231963 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.231964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.233418 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.233421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.238136 (+ 4715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.238137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.238307 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.238315 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 00:58:00.238511 (+ 196us) client_negotiation.cc:240] Negotiation successful
0806 00:58:00.238573 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":346,"thread_start_us":76,"threads_started":1}
I20260806 00:58:00.238884 26832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.229175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34164 (local address 127.23.242.1:36631)
0806 00:58:00.229392 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:00.229396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:00.229762 (+ 366us) server_negotiation.cc:407] Connection header received
0806 00:58:00.229815 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:00.229817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:00.229837 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:00.229920 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:00.230553 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.231851 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.233520 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.237775 (+ 4255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.237956 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.238040 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:00.238698 (+ 658us) server_negotiation.cc:299] Negotiation successful
0806 00:58:00.238755 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":81,"threads_started":1}
I20260806 00:58:00.239342 26836 tablet_bootstrap.cc:492] T 2dfcce24f52c4309b13154a54335376d P 4650ad0a2c9f4d308af8035b8b5efeb7: Bootstrap starting.
I20260806 00:58:00.239472 26835 tablet_bootstrap.cc:654] T 2dfcce24f52c4309b13154a54335376d P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:00.239835 26835 log.cc:824] T 2dfcce24f52c4309b13154a54335376d P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:00.240120 26836 tablet_bootstrap.cc:654] T 2dfcce24f52c4309b13154a54335376d P 4650ad0a2c9f4d308af8035b8b5efeb7: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:00.239988 26457 tablet_service.cc:1467] Processing CreateTablet for tablet 2dfcce24f52c4309b13154a54335376d (DEFAULT_TABLE table=test-table [id=87a79ddf4b384256b19120ad2616c765]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:00.240489 26836 log.cc:824] T 2dfcce24f52c4309b13154a54335376d P 4650ad0a2c9f4d308af8035b8b5efeb7: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:00.240592 26457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dfcce24f52c4309b13154a54335376d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:00.240787 26835 tablet_bootstrap.cc:492] T 2dfcce24f52c4309b13154a54335376d P 6e1948e9fd0b4766b3d8ac356d9d9fbc: No bootstrap required, opened a new log
I20260806 00:58:00.240860 26835 ts_tablet_manager.cc:1397] T 2dfcce24f52c4309b13154a54335376d P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 00:58:00.241285 26836 tablet_bootstrap.cc:492] T 2dfcce24f52c4309b13154a54335376d P 4650ad0a2c9f4d308af8035b8b5efeb7: No bootstrap required, opened a new log
I20260806 00:58:00.241520 26836 ts_tablet_manager.cc:1397] T 2dfcce24f52c4309b13154a54335376d P 4650ad0a2c9f4d308af8035b8b5efeb7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 00:58:00.243391 26836 raft_consensus.cc:348] T 2dfcce24f52c4309b13154a54335376d P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } }
I20260806 00:58:00.243564 26836 raft_consensus.cc:374] T 2dfcce24f52c4309b13154a54335376d P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:00.243610 26836 raft_consensus.cc:729] T 2dfcce24f52c4309b13154a54335376d P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4650ad0a2c9f4d308af8035b8b5efeb7, State: Initialized, Role: FOLLOWER
I20260806 00:58:00.243469 26835 raft_consensus.cc:348] T 2dfcce24f52c4309b13154a54335376d P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } }
I20260806 00:58:00.243654 26835 raft_consensus.cc:374] T 2dfcce24f52c4309b13154a54335376d P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:00.243712 26835 raft_consensus.cc:729] T 2dfcce24f52c4309b13154a54335376d P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e1948e9fd0b4766b3d8ac356d9d9fbc, State: Initialized, Role: FOLLOWER
I20260806 00:58:00.243733 26836 consensus_queue.cc:260] T 2dfcce24f52c4309b13154a54335376d P 4650ad0a2c9f4d308af8035b8b5efeb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } }
I20260806 00:58:00.243868 26835 consensus_queue.cc:260] T 2dfcce24f52c4309b13154a54335376d P 6e1948e9fd0b4766b3d8ac356d9d9fbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } }
I20260806 00:58:00.244100 26666 heartbeater.cc:503] Master 127.23.242.62:41397 was elected leader, sending a full tablet report...
I20260806 00:58:00.244222 26835 ts_tablet_manager.cc:1428] T 2dfcce24f52c4309b13154a54335376d P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260806 00:58:00.244264 26812 heartbeater.cc:503] Master 127.23.242.62:41397 was elected leader, sending a full tablet report...
I20260806 00:58:00.244032 26836 ts_tablet_manager.cc:1428] T 2dfcce24f52c4309b13154a54335376d P 4650ad0a2c9f4d308af8035b8b5efeb7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 00:58:00.245344 26841 tablet_bootstrap.cc:492] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4: Bootstrap starting.
W20260806 00:58:00.245846 26667 tablet.cc:2369] T 2dfcce24f52c4309b13154a54335376d P 4650ad0a2c9f4d308af8035b8b5efeb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 00:58:00.246150 26841 tablet_bootstrap.cc:654] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:00.246438 26841 log.cc:824] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:00.247282 26841 tablet_bootstrap.cc:492] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4: No bootstrap required, opened a new log
I20260806 00:58:00.247344 26841 ts_tablet_manager.cc:1397] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260806 00:58:00.248756 26841 raft_consensus.cc:348] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } }
I20260806 00:58:00.249161 26841 raft_consensus.cc:374] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:00.249293 26841 raft_consensus.cc:729] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e57237ca962b4e81a69d191e8d9c1ab4, State: Initialized, Role: FOLLOWER
I20260806 00:58:00.249622 26841 consensus_queue.cc:260] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } }
I20260806 00:58:00.250113 26841 ts_tablet_manager.cc:1428] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260806 00:58:00.250170 26525 heartbeater.cc:503] Master 127.23.242.62:41397 was elected leader, sending a full tablet report...
W20260806 00:58:00.307530 26526 tablet.cc:2369] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 00:58:00.423695 26843 raft_consensus.cc:482] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 00:58:00.423897 26843 raft_consensus.cc:504] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } }
I20260806 00:58:00.424402 26843 leader_election.cc:288] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e1948e9fd0b4766b3d8ac356d9d9fbc (127.23.242.3:39607), 4650ad0a2c9f4d308af8035b8b5efeb7 (127.23.242.2:33789)
I20260806 00:58:00.426973 26840 raft_consensus.cc:482] T 2dfcce24f52c4309b13154a54335376d P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 00:58:00.427551 26840 raft_consensus.cc:504] T 2dfcce24f52c4309b13154a54335376d P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } }
I20260806 00:58:00.428709 26840 leader_election.cc:288] T 2dfcce24f52c4309b13154a54335376d P 6e1948e9fd0b4766b3d8ac356d9d9fbc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4650ad0a2c9f4d308af8035b8b5efeb7 (127.23.242.2:33789), e57237ca962b4e81a69d191e8d9c1ab4 (127.23.242.1:36631)
I20260806 00:58:00.429040 26831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.424809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:52167 (local address 127.23.242.2:33789)
0806 00:58:00.424899 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:00.424902 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:00.424943 (+ 41us) server_negotiation.cc:407] Connection header received
0806 00:58:00.425004 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:00.425005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:00.425015 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:00.425075 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:00.425299 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.426140 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.428019 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.428606 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.428682 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.428693 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:00.428912 (+ 219us) server_negotiation.cc:299] Negotiation successful
0806 00:58:00.428955 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 00:58:00.429263 26844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.424649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:39607 (local address 127.23.242.1:55957)
0806 00:58:00.424822 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:00.424835 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:00.424903 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:00.425092 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:00.425093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:00.425095 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:00.425215 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.425218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.426296 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.426297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.427881 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.427886 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.428900 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.428901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.429004 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.429010 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 00:58:00.429066 (+ 56us) client_negotiation.cc:240] Negotiation successful
0806 00:58:00.429126 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":97,"thread_start_us":83,"threads_started":1}
I20260806 00:58:00.429543 26833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.424850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:55957 (local address 127.23.242.3:39607)
0806 00:58:00.424907 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:00.424909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:00.424953 (+ 44us) server_negotiation.cc:407] Connection header received
0806 00:58:00.424989 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:00.424990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:00.425001 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:00.425067 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:00.425306 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.426167 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.427989 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.428691 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.428822 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.428834 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:00.429432 (+ 598us) server_negotiation.cc:299] Negotiation successful
0806 00:58:00.429478 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 00:58:00.429579 26845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.424663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:33789 (local address 127.23.242.1:52167)
0806 00:58:00.424877 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:00.424886 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:00.424944 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:00.425092 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:00.425093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:00.425097 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:00.425178 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.425180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.426292 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.426294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.427889 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.427890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.428735 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.428736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.428820 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.428825 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 00:58:00.429427 (+ 602us) client_negotiation.cc:240] Negotiation successful
0806 00:58:00.429478 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":127,"thread_start_us":77,"threads_started":1}
I20260806 00:58:00.429857 26764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dfcce24f52c4309b13154a54335376d" candidate_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" is_pre_election: true
I20260806 00:58:00.429965 26764 raft_consensus.cc:2454] T 2dfcce24f52c4309b13154a54335376d P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e57237ca962b4e81a69d191e8d9c1ab4 in term 0.
I20260806 00:58:00.430184 26410 leader_election.cc:302] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e1948e9fd0b4766b3d8ac356d9d9fbc, e57237ca962b4e81a69d191e8d9c1ab4; no voters:
I20260806 00:58:00.430200 26620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dfcce24f52c4309b13154a54335376d" candidate_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" is_pre_election: true
I20260806 00:58:00.430347 26620 raft_consensus.cc:2454] T 2dfcce24f52c4309b13154a54335376d P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e57237ca962b4e81a69d191e8d9c1ab4 in term 0.
I20260806 00:58:00.430526 26843 raft_consensus.cc:2781] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 00:58:00.430675 26843 raft_consensus.cc:482] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 00:58:00.430764 26843 raft_consensus.cc:3037] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:00.431754 26843 raft_consensus.cc:504] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } }
I20260806 00:58:00.432086 26764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dfcce24f52c4309b13154a54335376d" candidate_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc"
I20260806 00:58:00.432181 26764 raft_consensus.cc:3037] T 2dfcce24f52c4309b13154a54335376d P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:00.432171 26620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dfcce24f52c4309b13154a54335376d" candidate_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7"
I20260806 00:58:00.432255 26620 raft_consensus.cc:3037] T 2dfcce24f52c4309b13154a54335376d P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:00.432327 26701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.428407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:36631 (local address 127.23.242.3:50301)
0806 00:58:00.428867 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:00.428878 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:00.428920 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:00.429110 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:00.429111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:00.429112 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:00.429167 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.429168 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.430187 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.430188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.431336 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.431340 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.432091 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.432092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.432179 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.432181 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 00:58:00.432226 (+ 45us) client_negotiation.cc:240] Negotiation successful
0806 00:58:00.432260 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":130,"threads_started":1}
I20260806 00:58:00.432646 26846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.428425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:34797 (local address 127.23.242.2:33789)
0806 00:58:00.428793 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:00.428796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:00.428803 (+ 7us) server_negotiation.cc:407] Connection header received
0806 00:58:00.428824 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:00.428824 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:00.428834 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:00.429079 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:00.429111 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.429809 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.431360 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.431957 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.432374 (+ 417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.432386 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:00.432406 (+ 20us) server_negotiation.cc:299] Negotiation successful
0806 00:58:00.432445 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":117,"threads_started":1}
I20260806 00:58:00.432652 26832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.428640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:50301 (local address 127.23.242.1:36631)
0806 00:58:00.428685 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:00.428687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:00.428975 (+ 288us) server_negotiation.cc:407] Connection header received
0806 00:58:00.429000 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:00.429001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:00.429010 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:00.429063 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:00.429242 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.429975 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.431452 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.431988 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.432055 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.432063 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:00.432420 (+ 357us) server_negotiation.cc:299] Negotiation successful
0806 00:58:00.432459 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":23}
I20260806 00:58:00.432328 26847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.428311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:33789 (local address 127.23.242.3:34797)
0806 00:58:00.428637 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:00.428649 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:00.428694 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:00.428936 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:00.428937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:00.428939 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:00.429001 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.429003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.429913 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.429915 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.431268 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.431271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.432062 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.432063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.432136 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.432140 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 00:58:00.432173 (+ 33us) client_negotiation.cc:240] Negotiation successful
0806 00:58:00.432217 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302}
I20260806 00:58:00.433379 26764 raft_consensus.cc:2454] T 2dfcce24f52c4309b13154a54335376d P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e57237ca962b4e81a69d191e8d9c1ab4 in term 1.
I20260806 00:58:00.433490 26478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dfcce24f52c4309b13154a54335376d" candidate_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" is_pre_election: true
I20260806 00:58:00.433571 26410 leader_election.cc:302] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e1948e9fd0b4766b3d8ac356d9d9fbc, e57237ca962b4e81a69d191e8d9c1ab4; no voters:
I20260806 00:58:00.433621 26478 raft_consensus.cc:2379] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e1948e9fd0b4766b3d8ac356d9d9fbc in current term 1: Already voted for candidate e57237ca962b4e81a69d191e8d9c1ab4 in this term.
I20260806 00:58:00.433755 26620 raft_consensus.cc:2454] T 2dfcce24f52c4309b13154a54335376d P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e57237ca962b4e81a69d191e8d9c1ab4 in term 1.
I20260806 00:58:00.433781 26849 raft_consensus.cc:2781] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:00.433903 26849 raft_consensus.cc:686] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4 [term 1 LEADER]: Becoming Leader. State: Replica: e57237ca962b4e81a69d191e8d9c1ab4, State: Running, Role: LEADER
I20260806 00:58:00.433905 26619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dfcce24f52c4309b13154a54335376d" candidate_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" is_pre_election: true
I20260806 00:58:00.434005 26619 raft_consensus.cc:2379] T 2dfcce24f52c4309b13154a54335376d P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e1948e9fd0b4766b3d8ac356d9d9fbc in current term 1: Already voted for candidate e57237ca962b4e81a69d191e8d9c1ab4 in this term.
I20260806 00:58:00.434005 26849 consensus_queue.cc:237] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } }
I20260806 00:58:00.434257 26695 leader_election.cc:302] T 2dfcce24f52c4309b13154a54335376d P 6e1948e9fd0b4766b3d8ac356d9d9fbc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e1948e9fd0b4766b3d8ac356d9d9fbc; no voters: 4650ad0a2c9f4d308af8035b8b5efeb7, e57237ca962b4e81a69d191e8d9c1ab4
W20260806 00:58:00.434386 26813 tablet.cc:2369] T 2dfcce24f52c4309b13154a54335376d P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 00:58:00.434422 26840 raft_consensus.cc:2726] T 2dfcce24f52c4309b13154a54335376d P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 00:58:00.434875 26843 leader_election.cc:288] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4 [CANDIDATE]: Term 1 election: Requested vote from peers 6e1948e9fd0b4766b3d8ac356d9d9fbc (127.23.242.3:39607), 4650ad0a2c9f4d308af8035b8b5efeb7 (127.23.242.2:33789)
I20260806 00:58:00.435088 26317 catalog_manager.cc:5495] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e57237ca962b4e81a69d191e8d9c1ab4 (127.23.242.1). New cstate: current_term: 1 leader_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } health_report { overall_health: HEALTHY } } }
Aug 06 00:58:00 dist-test-slave-dhph krb5kdc[26273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977879, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:00.485621 26859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.475604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:41397 (local address 127.0.0.1:48824)
0806 00:58:00.475816 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:00.475829 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:00.475927 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:00.476082 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:00.476083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:00.476688 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:00.476747 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.476750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.478026 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.478028 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.478930 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.478934 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.479376 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.479378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.479433 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.480350 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:00.480356 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:00.482282 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:00.484270 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:00.484280 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:00.484281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:00.484500 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:00.484789 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:00.484790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:00.484790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:00.484862 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:00.485242 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:00.485252 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:00.485310 (+ 58us) client_negotiation.cc:769] Sending connection context
0806 00:58:00.485379 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 00:58:00.485502 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":100,"threads_started":1}
I20260806 00:58:00.485639 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.475679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48824 (local address 127.23.242.62:41397)
0806 00:58:00.475780 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:00.475782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:00.475907 (+ 125us) server_negotiation.cc:407] Connection header received
0806 00:58:00.475993 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:00.475994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:00.476010 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:00.476077 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:00.477059 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.477767 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.479075 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.479276 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.479375 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.482419 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:00.482420 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:00.482421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:00.482439 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:00.484169 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:00.484602 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:00.484603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:00.484603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:00.484671 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:00.485004 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:00.485005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:00.485006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:00.485142 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:00.485201 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:00.485445 (+ 244us) server_negotiation.cc:299] Negotiation successful
0806 00:58:00.485559 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":66}
I20260806 00:58:00.488420 26317 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:48824:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 00:58:00.488626 26317 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 00:58:00.490510 26457 tablet_service.cc:1467] Processing CreateTablet for tablet daa8e74dbdb540418de902257a3c25ea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5692d3b58802459c91f35f827ff8ec6c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 00:58:00.490499 26599 tablet_service.cc:1467] Processing CreateTablet for tablet daa8e74dbdb540418de902257a3c25ea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5692d3b58802459c91f35f827ff8ec6c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 00:58:00.490607 26744 tablet_service.cc:1467] Processing CreateTablet for tablet daa8e74dbdb540418de902257a3c25ea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5692d3b58802459c91f35f827ff8ec6c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 00:58:00.490734 26457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daa8e74dbdb540418de902257a3c25ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:00.490734 26599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daa8e74dbdb540418de902257a3c25ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:00.490760 26744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daa8e74dbdb540418de902257a3c25ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:00.491966 26836 tablet_bootstrap.cc:492] T daa8e74dbdb540418de902257a3c25ea P 4650ad0a2c9f4d308af8035b8b5efeb7: Bootstrap starting.
I20260806 00:58:00.492269 26835 tablet_bootstrap.cc:492] T daa8e74dbdb540418de902257a3c25ea P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Bootstrap starting.
I20260806 00:58:00.492524 26836 tablet_bootstrap.cc:654] T daa8e74dbdb540418de902257a3c25ea P 4650ad0a2c9f4d308af8035b8b5efeb7: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:00.492551 26841 tablet_bootstrap.cc:492] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4: Bootstrap starting.
I20260806 00:58:00.492762 26835 tablet_bootstrap.cc:654] T daa8e74dbdb540418de902257a3c25ea P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:00.493121 26841 tablet_bootstrap.cc:654] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:00.493551 26836 tablet_bootstrap.cc:492] T daa8e74dbdb540418de902257a3c25ea P 4650ad0a2c9f4d308af8035b8b5efeb7: No bootstrap required, opened a new log
I20260806 00:58:00.493600 26836 ts_tablet_manager.cc:1397] T daa8e74dbdb540418de902257a3c25ea P 4650ad0a2c9f4d308af8035b8b5efeb7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260806 00:58:00.493922 26836 raft_consensus.cc:348] T daa8e74dbdb540418de902257a3c25ea P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:00.494025 26836 raft_consensus.cc:374] T daa8e74dbdb540418de902257a3c25ea P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:00.494058 26836 raft_consensus.cc:729] T daa8e74dbdb540418de902257a3c25ea P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4650ad0a2c9f4d308af8035b8b5efeb7, State: Initialized, Role: FOLLOWER
I20260806 00:58:00.494102 26836 consensus_queue.cc:260] T daa8e74dbdb540418de902257a3c25ea P 4650ad0a2c9f4d308af8035b8b5efeb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:00.494197 26836 ts_tablet_manager.cc:1428] T daa8e74dbdb540418de902257a3c25ea P 4650ad0a2c9f4d308af8035b8b5efeb7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:00.494375 26839 tablet_replica.cc:439] T daa8e74dbdb540418de902257a3c25ea P 4650ad0a2c9f4d308af8035b8b5efeb7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } } }
I20260806 00:58:00.494454 26839 tablet_replica.cc:442] T daa8e74dbdb540418de902257a3c25ea P 4650ad0a2c9f4d308af8035b8b5efeb7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 00:58:00.494499 26841 tablet_bootstrap.cc:492] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4: No bootstrap required, opened a new log
I20260806 00:58:00.494616 26841 ts_tablet_manager.cc:1397] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260806 00:58:00.494783 26835 tablet_bootstrap.cc:492] T daa8e74dbdb540418de902257a3c25ea P 6e1948e9fd0b4766b3d8ac356d9d9fbc: No bootstrap required, opened a new log
I20260806 00:58:00.494856 26835 ts_tablet_manager.cc:1397] T daa8e74dbdb540418de902257a3c25ea P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260806 00:58:00.495034 26835 raft_consensus.cc:348] T daa8e74dbdb540418de902257a3c25ea P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:00.495065 26841 raft_consensus.cc:348] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:00.495162 26841 raft_consensus.cc:374] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:00.495115 26835 raft_consensus.cc:374] T daa8e74dbdb540418de902257a3c25ea P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:00.495189 26841 raft_consensus.cc:729] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e57237ca962b4e81a69d191e8d9c1ab4, State: Initialized, Role: FOLLOWER
I20260806 00:58:00.495213 26835 raft_consensus.cc:729] T daa8e74dbdb540418de902257a3c25ea P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e1948e9fd0b4766b3d8ac356d9d9fbc, State: Initialized, Role: FOLLOWER
I20260806 00:58:00.495231 26841 consensus_queue.cc:260] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:00.495335 26841 ts_tablet_manager.cc:1428] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 00:58:00.495299 26835 consensus_queue.cc:260] T daa8e74dbdb540418de902257a3c25ea P 6e1948e9fd0b4766b3d8ac356d9d9fbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:00.495416 26835 ts_tablet_manager.cc:1428] T daa8e74dbdb540418de902257a3c25ea P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:00.495523 26840 tablet_replica.cc:439] T daa8e74dbdb540418de902257a3c25ea P 6e1948e9fd0b4766b3d8ac356d9d9fbc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } } }
I20260806 00:58:00.495535 26843 tablet_replica.cc:439] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } } }
I20260806 00:58:00.495623 26840 tablet_replica.cc:442] T daa8e74dbdb540418de902257a3c25ea P 6e1948e9fd0b4766b3d8ac356d9d9fbc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 00:58:00.495642 26843 tablet_replica.cc:442] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 00:58:00.563874 26839 raft_consensus.cc:482] T daa8e74dbdb540418de902257a3c25ea P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 00:58:00.563980 26839 raft_consensus.cc:504] T daa8e74dbdb540418de902257a3c25ea P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:00.564594 26839 leader_election.cc:288] T daa8e74dbdb540418de902257a3c25ea P 4650ad0a2c9f4d308af8035b8b5efeb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e57237ca962b4e81a69d191e8d9c1ab4 (127.23.242.1:36631), 6e1948e9fd0b4766b3d8ac356d9d9fbc (127.23.242.3:39607)
I20260806 00:58:00.565555 26843 raft_consensus.cc:482] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 00:58:00.565644 26843 raft_consensus.cc:504] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:00.565789 26843 leader_election.cc:288] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4650ad0a2c9f4d308af8035b8b5efeb7 (127.23.242.2:33789), 6e1948e9fd0b4766b3d8ac356d9d9fbc (127.23.242.3:39607)
I20260806 00:58:00.565951 26764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa8e74dbdb540418de902257a3c25ea" candidate_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" is_pre_election: true
I20260806 00:58:00.566036 26764 raft_consensus.cc:2454] T daa8e74dbdb540418de902257a3c25ea P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e57237ca962b4e81a69d191e8d9c1ab4 in term 0.
I20260806 00:58:00.566156 26410 leader_election.cc:302] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e1948e9fd0b4766b3d8ac356d9d9fbc, e57237ca962b4e81a69d191e8d9c1ab4; no voters:
I20260806 00:58:00.566305 26843 raft_consensus.cc:2781] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 00:58:00.566355 26843 raft_consensus.cc:482] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 00:58:00.566382 26843 raft_consensus.cc:3037] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:00.566654 26619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa8e74dbdb540418de902257a3c25ea" candidate_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" is_pre_election: true
I20260806 00:58:00.566746 26619 raft_consensus.cc:2454] T daa8e74dbdb540418de902257a3c25ea P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e57237ca962b4e81a69d191e8d9c1ab4 in term 0.
I20260806 00:58:00.567276 26843 raft_consensus.cc:504] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:00.567399 26843 leader_election.cc:288] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4 [CANDIDATE]: Term 1 election: Requested vote from peers 4650ad0a2c9f4d308af8035b8b5efeb7 (127.23.242.2:33789), 6e1948e9fd0b4766b3d8ac356d9d9fbc (127.23.242.3:39607)
I20260806 00:58:00.567566 26619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa8e74dbdb540418de902257a3c25ea" candidate_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7"
I20260806 00:58:00.567643 26619 raft_consensus.cc:3037] T daa8e74dbdb540418de902257a3c25ea P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:00.567622 26764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa8e74dbdb540418de902257a3c25ea" candidate_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc"
I20260806 00:58:00.567689 26764 raft_consensus.cc:3037] T daa8e74dbdb540418de902257a3c25ea P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:00.568445 26764 raft_consensus.cc:2454] T daa8e74dbdb540418de902257a3c25ea P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e57237ca962b4e81a69d191e8d9c1ab4 in term 1.
I20260806 00:58:00.568552 26833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.564975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:58511 (local address 127.23.242.3:39607)
0806 00:58:00.565030 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:00.565032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:00.565046 (+ 14us) server_negotiation.cc:407] Connection header received
0806 00:58:00.565157 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:00.565158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:00.565170 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:00.565254 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:00.565401 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.566073 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.567579 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.568086 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.568149 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.568160 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:00.568409 (+ 249us) server_negotiation.cc:299] Negotiation successful
0806 00:58:00.568474 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 00:58:00.568567 26619 raft_consensus.cc:2454] T daa8e74dbdb540418de902257a3c25ea P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e57237ca962b4e81a69d191e8d9c1ab4 in term 1.
I20260806 00:58:00.568589 26410 leader_election.cc:302] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e1948e9fd0b4766b3d8ac356d9d9fbc, e57237ca962b4e81a69d191e8d9c1ab4; no voters:
I20260806 00:58:00.568674 26843 raft_consensus.cc:2781] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:00.568709 26843 raft_consensus.cc:686] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4 [term 1 LEADER]: Becoming Leader. State: Replica: e57237ca962b4e81a69d191e8d9c1ab4, State: Running, Role: LEADER
I20260806 00:58:00.568748 26843 consensus_queue.cc:237] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:00.568825 26863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.564834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:39607 (local address 127.23.242.2:58511)
0806 00:58:00.564984 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:00.564997 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:00.565066 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:00.565262 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:00.565263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:00.565265 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:00.565330 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.565332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.566182 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.566183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.567477 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.567480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.568227 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.568227 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.568309 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.568313 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 00:58:00.568689 (+ 376us) client_negotiation.cc:240] Negotiation successful
0806 00:58:00.568744 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":105,"threads_started":1}
I20260806 00:58:00.568990 26832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.564904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:35011 (local address 127.23.242.1:36631)
0806 00:58:00.564961 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:00.564964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:00.565259 (+ 295us) server_negotiation.cc:407] Connection header received
0806 00:58:00.565339 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:00.565340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:00.565351 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:00.565406 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:00.565569 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.566378 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.567830 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.568450 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.568508 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.568518 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:00.568854 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 00:58:00.568901 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 00:58:00.569036 26843 tablet_replica.cc:439] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4: TxnStatusTablet state changed. Reason: New leader e57237ca962b4e81a69d191e8d9c1ab4. Latest consensus state: current_term: 1 leader_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } } }
I20260806 00:58:00.569114 26843 tablet_replica.cc:442] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4: This TxnStatusTablet replica's current role is: LEADER
I20260806 00:58:00.569308 26864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.564837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:36631 (local address 127.23.242.2:35011)
0806 00:58:00.565163 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:00.565174 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:00.565274 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:00.565421 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:00.565422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:00.565424 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:00.565504 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.565506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.566569 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.566570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.567704 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.567707 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.568591 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.568592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.568760 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.568764 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 00:58:00.569163 (+ 399us) client_negotiation.cc:240] Negotiation successful
0806 00:58:00.569218 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":286,"threads_started":1}
I20260806 00:58:00.569387 26764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa8e74dbdb540418de902257a3c25ea" candidate_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" is_pre_election: true
I20260806 00:58:00.569482 26764 raft_consensus.cc:2379] T daa8e74dbdb540418de902257a3c25ea P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4650ad0a2c9f4d308af8035b8b5efeb7 in current term 1: Already voted for candidate e57237ca962b4e81a69d191e8d9c1ab4 in this term.
I20260806 00:58:00.569512 26478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa8e74dbdb540418de902257a3c25ea" candidate_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" is_pre_election: true
I20260806 00:58:00.569576 26866 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 00:58:00.569722 26866 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 00:58:00.569761 26552 leader_election.cc:302] T daa8e74dbdb540418de902257a3c25ea P 4650ad0a2c9f4d308af8035b8b5efeb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4650ad0a2c9f4d308af8035b8b5efeb7; no voters: 6e1948e9fd0b4766b3d8ac356d9d9fbc, e57237ca962b4e81a69d191e8d9c1ab4
I20260806 00:58:00.569867 26839 raft_consensus.cc:2726] T daa8e74dbdb540418de902257a3c25ea P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 00:58:00.569932 26317 catalog_manager.cc:5495] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e57237ca962b4e81a69d191e8d9c1ab4 (127.23.242.1). New cstate: current_term: 1 leader_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } health_report { overall_health: UNKNOWN } } }
I20260806 00:58:00.595216 26859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.591151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:36631 (local address 127.0.0.1:34178)
0806 00:58:00.591250 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:00.591266 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:00.591335 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:00.591755 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:00.591757 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:00.591758 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 00:58:00.591827 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.591830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.592768 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.592769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.593936 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.593940 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.594485 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.594487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.594567 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.594633 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 00:58:00.594978 (+ 345us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 00:58:00.594991 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 00:58:00.595058 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 00:58:00.595119 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 00:58:00.595247 26832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.591450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34178 (local address 127.23.242.1:36631)
0806 00:58:00.591551 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:00.591553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:00.591565 (+ 12us) server_negotiation.cc:407] Connection header received
0806 00:58:00.591611 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:00.591612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:00.591636 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:00.591719 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 00:58:00.591918 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.592662 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.594064 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.594384 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.594470 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.594788 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 00:58:00.594893 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 00:58:00.594964 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:00.595156 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 00:58:00.595179 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 00:58:00.600682 26764 raft_consensus.cc:1261] T daa8e74dbdb540418de902257a3c25ea P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 1 FOLLOWER]: Refusing update from remote peer e57237ca962b4e81a69d191e8d9c1ab4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:00.600694 26619 raft_consensus.cc:1261] T daa8e74dbdb540418de902257a3c25ea P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 1 FOLLOWER]: Refusing update from remote peer e57237ca962b4e81a69d191e8d9c1ab4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:00.600996 26849 consensus_queue.cc:1035] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:00.602782 26839 tablet_replica.cc:439] T daa8e74dbdb540418de902257a3c25ea P 4650ad0a2c9f4d308af8035b8b5efeb7: TxnStatusTablet state changed. Reason: New leader e57237ca962b4e81a69d191e8d9c1ab4. Latest consensus state: current_term: 1 leader_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } } }
I20260806 00:58:00.602893 26839 tablet_replica.cc:442] T daa8e74dbdb540418de902257a3c25ea P 4650ad0a2c9f4d308af8035b8b5efeb7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 00:58:00.603559 26849 consensus_queue.cc:1035] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:00.604212 26849 tablet_replica.cc:439] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } } }
I20260806 00:58:00.604298 26849 tablet_replica.cc:442] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4: This TxnStatusTablet replica's current role is: LEADER
I20260806 00:58:00.604998 26840 tablet_replica.cc:439] T daa8e74dbdb540418de902257a3c25ea P 6e1948e9fd0b4766b3d8ac356d9d9fbc: TxnStatusTablet state changed. Reason: New leader e57237ca962b4e81a69d191e8d9c1ab4. Latest consensus state: current_term: 1 leader_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } } }
I20260806 00:58:00.605041 26870 mvcc.cc:205] Tried to move back new op lower bound from 7315365398934822912 to 7315365398809845760. Current Snapshot: MvccSnapshot[applied={T|T < 7315365398934822912}]
I20260806 00:58:00.605094 26840 tablet_replica.cc:442] T daa8e74dbdb540418de902257a3c25ea P 6e1948e9fd0b4766b3d8ac356d9d9fbc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 00:58:00.605141 26839 tablet_replica.cc:439] T daa8e74dbdb540418de902257a3c25ea P 4650ad0a2c9f4d308af8035b8b5efeb7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } } }
I20260806 00:58:00.605224 26839 tablet_replica.cc:442] T daa8e74dbdb540418de902257a3c25ea P 4650ad0a2c9f4d308af8035b8b5efeb7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 00:58:00.606526 26843 tablet_replica.cc:439] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } } }
I20260806 00:58:00.606654 26843 tablet_replica.cc:442] T daa8e74dbdb540418de902257a3c25ea P e57237ca962b4e81a69d191e8d9c1ab4: This TxnStatusTablet replica's current role is: LEADER
I20260806 00:58:00.606972 26840 tablet_replica.cc:439] T daa8e74dbdb540418de902257a3c25ea P 6e1948e9fd0b4766b3d8ac356d9d9fbc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } } }
I20260806 00:58:00.607038 26874 mvcc.cc:205] Tried to move back new op lower bound from 7315365398934822912 to 7315365398809845760. Current Snapshot: MvccSnapshot[applied={T|T < 7315365398934822912}]
I20260806 00:58:00.607071 26840 tablet_replica.cc:442] T daa8e74dbdb540418de902257a3c25ea P 6e1948e9fd0b4766b3d8ac356d9d9fbc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 00:58:00.610452 26832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.607958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34186 (local address 127.23.242.1:36631)
0806 00:58:00.608073 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:00.608075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:00.608081 (+ 6us) server_negotiation.cc:407] Connection header received
0806 00:58:00.608107 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:00.608108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:00.608138 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:00.608198 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 00:58:00.608409 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.608894 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.609674 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.609892 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.609969 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.610140 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 00:58:00.610174 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 00:58:00.610222 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:00.610340 (+ 118us) server_negotiation.cc:299] Negotiation successful
0806 00:58:00.610364 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 00:58:00.616766 26832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.613809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:51275 (local address 127.23.242.1:36631)
0806 00:58:00.613983 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:00.613986 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:00.614163 (+ 177us) server_negotiation.cc:407] Connection header received
0806 00:58:00.614234 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:00.614235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:00.614245 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:00.614298 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 00:58:00.614455 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.614992 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.615943 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.616146 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.616230 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.616460 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 00:58:00.616499 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 00:58:00.616559 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:00.616613 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 00:58:00.616665 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":20}
I20260806 00:58:00.616786 26881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.613666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:36631 (local address 127.23.242.1:51275)
0806 00:58:00.614085 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:00.614095 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:00.614182 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:00.614307 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:00.614307 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:00.614309 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 00:58:00.614369 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.614371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.615069 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.615070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.615821 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.615824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.616254 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.616255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.616329 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.616345 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 00:58:00.616560 (+ 215us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 00:58:00.616561 (+ 1us) client_negotiation.cc:769] Sending connection context
0806 00:58:00.616609 (+ 48us) client_negotiation.cc:240] Negotiation successful
0806 00:58:00.616684 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":17,"thread_start_us":193,"threads_started":1}
I20260806 00:58:00.621109 26619 raft_consensus.cc:1261] T 2dfcce24f52c4309b13154a54335376d P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 1 FOLLOWER]: Refusing update from remote peer e57237ca962b4e81a69d191e8d9c1ab4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:00.621177 26764 raft_consensus.cc:1261] T 2dfcce24f52c4309b13154a54335376d P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 1 FOLLOWER]: Refusing update from remote peer e57237ca962b4e81a69d191e8d9c1ab4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:00.621483 26843 consensus_queue.cc:1035] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:00.622352 26850 consensus_queue.cc:1035] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:00.625097 26869 mvcc.cc:205] Tried to move back new op lower bound from 7315365399022137344 to 7315365398258458624. Current Snapshot: MvccSnapshot[applied={T|T < 7315365399022137344}]
I20260806 00:58:00.641268 26317 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:58:00.641410 26317 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:58:00.642437 26317 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70823b4c6d02429ca702661edb1041e8: Sending DeleteTablet for 3 replicas of tablet 2dfcce24f52c4309b13154a54335376d
I20260806 00:58:00.642794 26744 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dfcce24f52c4309b13154a54335376d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:00 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:39620
I20260806 00:58:00.642884 26599 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dfcce24f52c4309b13154a54335376d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:00 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:45398
I20260806 00:58:00.643088 26887 tablet_replica.cc:330] T 2dfcce24f52c4309b13154a54335376d P 6e1948e9fd0b4766b3d8ac356d9d9fbc: stopping tablet replica
I20260806 00:58:00.643314 26887 raft_consensus.cc:2229] T 2dfcce24f52c4309b13154a54335376d P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:00.643424 26887 raft_consensus.cc:2258] T 2dfcce24f52c4309b13154a54335376d P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:00.643363 26888 tablet_replica.cc:330] T 2dfcce24f52c4309b13154a54335376d P 4650ad0a2c9f4d308af8035b8b5efeb7: stopping tablet replica
I20260806 00:58:00.643692 26888 raft_consensus.cc:2229] T 2dfcce24f52c4309b13154a54335376d P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:00.643723 26457 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dfcce24f52c4309b13154a54335376d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:00 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:34164
I20260806 00:58:00.643859 26888 raft_consensus.cc:2258] T 2dfcce24f52c4309b13154a54335376d P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:00.643981 26889 tablet_replica.cc:330] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4: stopping tablet replica
I20260806 00:58:00.644088 26889 raft_consensus.cc:2229] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4 [term 1 LEADER]: Raft consensus shutting down.
I20260806 00:58:00.644035 26317 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 00:58:00.644181 26317 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 00:58:00.644209 26889 raft_consensus.cc:2258] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:00.644846 26887 ts_tablet_manager.cc:1905] T 2dfcce24f52c4309b13154a54335376d P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:00.645174 26888 ts_tablet_manager.cc:1905] T 2dfcce24f52c4309b13154a54335376d P 4650ad0a2c9f4d308af8035b8b5efeb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:00.645356 26889 ts_tablet_manager.cc:1905] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:00.646725 26887 ts_tablet_manager.cc:1918] T 2dfcce24f52c4309b13154a54335376d P 6e1948e9fd0b4766b3d8ac356d9d9fbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 00:58:00.646814 26887 log.cc:1196] T 2dfcce24f52c4309b13154a54335376d P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-2/wal/wals/2dfcce24f52c4309b13154a54335376d
I20260806 00:58:00.647120 26887 ts_tablet_manager.cc:1939] T 2dfcce24f52c4309b13154a54335376d P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Deleting consensus metadata
I20260806 00:58:00.647106 26599 tablet_service.cc:1467] Processing CreateTablet for tablet 5e195e456e2a4dfc953a21ab3f1dbeae (DEFAULT_TABLE table=test-table [id=88cd37fc0601404fb24222e9a1fc0208]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:00.647231 26599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e195e456e2a4dfc953a21ab3f1dbeae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:00.648279 26836 tablet_bootstrap.cc:492] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7: Bootstrap starting.
I20260806 00:58:00.648524 26304 catalog_manager.cc:4841] TS 6e1948e9fd0b4766b3d8ac356d9d9fbc (127.23.242.3:39607): tablet 2dfcce24f52c4309b13154a54335376d (table test-table [id=87a79ddf4b384256b19120ad2616c765]) successfully deleted
I20260806 00:58:00.648732 26836 tablet_bootstrap.cc:654] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:00.648761 26744 tablet_service.cc:1467] Processing CreateTablet for tablet 5e195e456e2a4dfc953a21ab3f1dbeae (DEFAULT_TABLE table=test-table [id=88cd37fc0601404fb24222e9a1fc0208]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:00.648954 26744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e195e456e2a4dfc953a21ab3f1dbeae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:00.649420 26836 tablet_bootstrap.cc:492] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7: No bootstrap required, opened a new log
I20260806 00:58:00.649477 26836 ts_tablet_manager.cc:1397] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:00.649633 26836 raft_consensus.cc:348] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:00.649725 26836 raft_consensus.cc:374] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:00.649758 26836 raft_consensus.cc:729] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4650ad0a2c9f4d308af8035b8b5efeb7, State: Initialized, Role: FOLLOWER
I20260806 00:58:00.649812 26836 consensus_queue.cc:260] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:00.649907 26836 ts_tablet_manager.cc:1428] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260806 00:58:00.650192 26889 ts_tablet_manager.cc:1918] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 00:58:00.650324 26835 tablet_bootstrap.cc:492] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Bootstrap starting.
I20260806 00:58:00.650499 26888 ts_tablet_manager.cc:1918] T 2dfcce24f52c4309b13154a54335376d P 4650ad0a2c9f4d308af8035b8b5efeb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 00:58:00.650656 26889 log.cc:1196] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-0/wal/wals/2dfcce24f52c4309b13154a54335376d
I20260806 00:58:00.650801 26835 tablet_bootstrap.cc:654] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:00.650926 26888 log.cc:1196] T 2dfcce24f52c4309b13154a54335376d P 4650ad0a2c9f4d308af8035b8b5efeb7: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-1/wal/wals/2dfcce24f52c4309b13154a54335376d
I20260806 00:58:00.651142 26457 tablet_service.cc:1467] Processing CreateTablet for tablet 5e195e456e2a4dfc953a21ab3f1dbeae (DEFAULT_TABLE table=test-table [id=88cd37fc0601404fb24222e9a1fc0208]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:00.651384 26457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e195e456e2a4dfc953a21ab3f1dbeae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:00.651521 26889 ts_tablet_manager.cc:1939] T 2dfcce24f52c4309b13154a54335376d P e57237ca962b4e81a69d191e8d9c1ab4: Deleting consensus metadata
I20260806 00:58:00.651670 26888 ts_tablet_manager.cc:1939] T 2dfcce24f52c4309b13154a54335376d P 4650ad0a2c9f4d308af8035b8b5efeb7: Deleting consensus metadata
I20260806 00:58:00.651738 26835 tablet_bootstrap.cc:492] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc: No bootstrap required, opened a new log
I20260806 00:58:00.651790 26835 ts_tablet_manager.cc:1397] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:00.651930 26835 raft_consensus.cc:348] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:00.652014 26835 raft_consensus.cc:374] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:00.652041 26835 raft_consensus.cc:729] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e1948e9fd0b4766b3d8ac356d9d9fbc, State: Initialized, Role: FOLLOWER
I20260806 00:58:00.652086 26835 consensus_queue.cc:260] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:00.652218 26835 ts_tablet_manager.cc:1428] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260806 00:58:00.652779 26305 catalog_manager.cc:4841] TS 4650ad0a2c9f4d308af8035b8b5efeb7 (127.23.242.2:33789): tablet 2dfcce24f52c4309b13154a54335376d (table test-table [id=87a79ddf4b384256b19120ad2616c765]) successfully deleted
I20260806 00:58:00.652872 26841 tablet_bootstrap.cc:492] T 5e195e456e2a4dfc953a21ab3f1dbeae P e57237ca962b4e81a69d191e8d9c1ab4: Bootstrap starting.
I20260806 00:58:00.653194 26302 catalog_manager.cc:4841] TS e57237ca962b4e81a69d191e8d9c1ab4 (127.23.242.1:36631): tablet 2dfcce24f52c4309b13154a54335376d (table test-table [id=87a79ddf4b384256b19120ad2616c765]) successfully deleted
I20260806 00:58:00.653322 26841 tablet_bootstrap.cc:654] T 5e195e456e2a4dfc953a21ab3f1dbeae P e57237ca962b4e81a69d191e8d9c1ab4: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:00.654022 26841 tablet_bootstrap.cc:492] T 5e195e456e2a4dfc953a21ab3f1dbeae P e57237ca962b4e81a69d191e8d9c1ab4: No bootstrap required, opened a new log
I20260806 00:58:00.654093 26841 ts_tablet_manager.cc:1397] T 5e195e456e2a4dfc953a21ab3f1dbeae P e57237ca962b4e81a69d191e8d9c1ab4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:00.654268 26841 raft_consensus.cc:348] T 5e195e456e2a4dfc953a21ab3f1dbeae P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:00.654332 26841 raft_consensus.cc:374] T 5e195e456e2a4dfc953a21ab3f1dbeae P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:00.654346 26841 raft_consensus.cc:729] T 5e195e456e2a4dfc953a21ab3f1dbeae P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e57237ca962b4e81a69d191e8d9c1ab4, State: Initialized, Role: FOLLOWER
I20260806 00:58:00.654402 26841 consensus_queue.cc:260] T 5e195e456e2a4dfc953a21ab3f1dbeae P e57237ca962b4e81a69d191e8d9c1ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:00.654502 26841 ts_tablet_manager.cc:1428] T 5e195e456e2a4dfc953a21ab3f1dbeae P e57237ca962b4e81a69d191e8d9c1ab4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:00.869447 26839 raft_consensus.cc:482] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 00:58:00.869567 26839 raft_consensus.cc:504] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:00.869777 26839 leader_election.cc:288] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e57237ca962b4e81a69d191e8d9c1ab4 (127.23.242.1:36631), 6e1948e9fd0b4766b3d8ac356d9d9fbc (127.23.242.3:39607)
I20260806 00:58:00.870231 26478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e195e456e2a4dfc953a21ab3f1dbeae" candidate_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" is_pre_election: true
I20260806 00:58:00.870317 26763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e195e456e2a4dfc953a21ab3f1dbeae" candidate_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" is_pre_election: true
I20260806 00:58:00.870406 26478 raft_consensus.cc:2454] T 5e195e456e2a4dfc953a21ab3f1dbeae P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4650ad0a2c9f4d308af8035b8b5efeb7 in term 0.
I20260806 00:58:00.870414 26763 raft_consensus.cc:2454] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4650ad0a2c9f4d308af8035b8b5efeb7 in term 0.
I20260806 00:58:00.870513 26840 raft_consensus.cc:482] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 00:58:00.870646 26550 leader_election.cc:302] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4650ad0a2c9f4d308af8035b8b5efeb7, e57237ca962b4e81a69d191e8d9c1ab4; no voters:
I20260806 00:58:00.870623 26840 raft_consensus.cc:504] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:00.870748 26839 raft_consensus.cc:2781] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 00:58:00.870782 26839 raft_consensus.cc:482] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 00:58:00.870783 26840 leader_election.cc:288] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e57237ca962b4e81a69d191e8d9c1ab4 (127.23.242.1:36631), 4650ad0a2c9f4d308af8035b8b5efeb7 (127.23.242.2:33789)
I20260806 00:58:00.870805 26839 raft_consensus.cc:3037] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:00.870991 26619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e195e456e2a4dfc953a21ab3f1dbeae" candidate_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" is_pre_election: true
I20260806 00:58:00.871101 26478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e195e456e2a4dfc953a21ab3f1dbeae" candidate_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" is_pre_election: true
I20260806 00:58:00.871192 26478 raft_consensus.cc:2454] T 5e195e456e2a4dfc953a21ab3f1dbeae P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e1948e9fd0b4766b3d8ac356d9d9fbc in term 0.
I20260806 00:58:00.871383 26692 leader_election.cc:302] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e1948e9fd0b4766b3d8ac356d9d9fbc, e57237ca962b4e81a69d191e8d9c1ab4; no voters:
I20260806 00:58:00.871546 26840 raft_consensus.cc:2781] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 00:58:00.871608 26840 raft_consensus.cc:482] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 00:58:00.871629 26840 raft_consensus.cc:3037] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:00.872205 26839 raft_consensus.cc:504] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:00.872387 26839 leader_election.cc:288] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7 [CANDIDATE]: Term 1 election: Requested vote from peers e57237ca962b4e81a69d191e8d9c1ab4 (127.23.242.1:36631), 6e1948e9fd0b4766b3d8ac356d9d9fbc (127.23.242.3:39607)
I20260806 00:58:00.872406 26619 raft_consensus.cc:2379] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e1948e9fd0b4766b3d8ac356d9d9fbc in current term 1: Already voted for candidate 4650ad0a2c9f4d308af8035b8b5efeb7 in this term.
I20260806 00:58:00.872597 26840 raft_consensus.cc:504] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:00.872643 26478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e195e456e2a4dfc953a21ab3f1dbeae" candidate_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e57237ca962b4e81a69d191e8d9c1ab4"
I20260806 00:58:00.872733 26840 leader_election.cc:288] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc [CANDIDATE]: Term 1 election: Requested vote from peers e57237ca962b4e81a69d191e8d9c1ab4 (127.23.242.1:36631), 4650ad0a2c9f4d308af8035b8b5efeb7 (127.23.242.2:33789)
I20260806 00:58:00.872735 26478 raft_consensus.cc:3037] T 5e195e456e2a4dfc953a21ab3f1dbeae P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:00.872804 26763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e195e456e2a4dfc953a21ab3f1dbeae" candidate_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc"
I20260806 00:58:00.872864 26763 raft_consensus.cc:2379] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4650ad0a2c9f4d308af8035b8b5efeb7 in current term 1: Already voted for candidate 6e1948e9fd0b4766b3d8ac356d9d9fbc in this term.
I20260806 00:58:00.872856 26619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e195e456e2a4dfc953a21ab3f1dbeae" candidate_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7"
I20260806 00:58:00.872920 26619 raft_consensus.cc:2379] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e1948e9fd0b4766b3d8ac356d9d9fbc in current term 1: Already voted for candidate 4650ad0a2c9f4d308af8035b8b5efeb7 in this term.
I20260806 00:58:00.872892 26477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e195e456e2a4dfc953a21ab3f1dbeae" candidate_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e57237ca962b4e81a69d191e8d9c1ab4"
I20260806 00:58:00.873138 26692 leader_election.cc:302] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e1948e9fd0b4766b3d8ac356d9d9fbc; no voters: 4650ad0a2c9f4d308af8035b8b5efeb7, e57237ca962b4e81a69d191e8d9c1ab4
I20260806 00:58:00.873238 26840 raft_consensus.cc:2726] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 00:58:00.873657 26478 raft_consensus.cc:2454] T 5e195e456e2a4dfc953a21ab3f1dbeae P e57237ca962b4e81a69d191e8d9c1ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4650ad0a2c9f4d308af8035b8b5efeb7 in term 1.
I20260806 00:58:00.873838 26550 leader_election.cc:302] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4650ad0a2c9f4d308af8035b8b5efeb7, e57237ca962b4e81a69d191e8d9c1ab4; no voters: 6e1948e9fd0b4766b3d8ac356d9d9fbc
I20260806 00:58:00.874009 26839 raft_consensus.cc:2781] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:00.874140 26839 raft_consensus.cc:686] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 1 LEADER]: Becoming Leader. State: Replica: 4650ad0a2c9f4d308af8035b8b5efeb7, State: Running, Role: LEADER
I20260806 00:58:00.874287 26839 consensus_queue.cc:237] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:00.875110 26317 catalog_manager.cc:5495] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4650ad0a2c9f4d308af8035b8b5efeb7 (127.23.242.2). New cstate: current_term: 1 leader_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } health_report { overall_health: UNKNOWN } } }
I20260806 00:58:00.889322 26846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.886824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45412 (local address 127.23.242.2:33789)
0806 00:58:00.886886 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:00.886889 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:00.886914 (+ 25us) server_negotiation.cc:407] Connection header received
0806 00:58:00.887005 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:00.887006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:00.887032 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:00.887143 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 00:58:00.887253 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.887740 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.888471 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.888703 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.888779 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.888926 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 00:58:00.889044 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 00:58:00.889134 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:00.889176 (+ 42us) server_negotiation.cc:299] Negotiation successful
0806 00:58:00.889201 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 00:58:00.895985 26897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.892652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:36631 (local address 127.23.242.2:36943)
0806 00:58:00.892949 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:00.892960 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:00.893020 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:00.893254 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:00.893255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:00.893257 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 00:58:00.893359 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.893360 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.894086 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.894087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.894799 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.894802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.895357 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.895358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.895429 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.895449 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 00:58:00.895786 (+ 337us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 00:58:00.895788 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 00:58:00.895838 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 00:58:00.895889 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":189,"threads_started":1}
I20260806 00:58:00.895977 26832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.892846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:36943 (local address 127.23.242.1:36631)
0806 00:58:00.892937 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:00.892939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:00.893060 (+ 121us) server_negotiation.cc:407] Connection header received
0806 00:58:00.893094 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:00.893095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:00.893106 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:00.893176 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 00:58:00.893459 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.893949 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.895059 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.895264 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.895329 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.895620 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 00:58:00.895654 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 00:58:00.895707 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:00.895885 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 00:58:00.895910 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 00:58:00.899045 26763 raft_consensus.cc:1261] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 1 FOLLOWER]: Refusing update from remote peer 4650ad0a2c9f4d308af8035b8b5efeb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:00.899112 26478 raft_consensus.cc:1261] T 5e195e456e2a4dfc953a21ab3f1dbeae P e57237ca962b4e81a69d191e8d9c1ab4 [term 1 FOLLOWER]: Refusing update from remote peer 4650ad0a2c9f4d308af8035b8b5efeb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:00.899335 26893 consensus_queue.cc:1035] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:00.899492 26839 consensus_queue.cc:1035] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:00.907846 26846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.905130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:44437 (local address 127.23.242.2:33789)
0806 00:58:00.905173 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:00.905175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:00.905217 (+ 42us) server_negotiation.cc:407] Connection header received
0806 00:58:00.905309 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:00.905311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:00.905329 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:00.905405 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 00:58:00.905519 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.906052 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.906982 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.907198 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.907260 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.907558 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 00:58:00.907600 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 00:58:00.907655 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:00.907727 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 00:58:00.907754 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 00:58:00.907883 26881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:00.905077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:33789 (local address 127.23.242.1:44437)
0806 00:58:00.905161 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:00.905182 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:00.905234 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:00.905410 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:00.905411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:00.905412 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 00:58:00.905462 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.905464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.906170 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.906171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.906857 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:00.906860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:00.907353 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:00.907355 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:00.907429 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:00.907438 (+ 9us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 00:58:00.907679 (+ 241us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 00:58:00.907683 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 00:58:00.907721 (+ 38us) client_negotiation.cc:240] Negotiation successful
0806 00:58:00.907789 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
W20260806 00:58:00.908289 26870 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 00:58:02.437773 26317 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:58:02.437918 26317 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:58:02.439065 26317 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70823b4c6d02429ca702661edb1041e8: Sending DeleteTablet for 3 replicas of tablet 5e195e456e2a4dfc953a21ab3f1dbeae
I20260806 00:58:02.439417 26599 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e195e456e2a4dfc953a21ab3f1dbeae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:02 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:45398
I20260806 00:58:02.439458 26744 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e195e456e2a4dfc953a21ab3f1dbeae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:02 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:39620
I20260806 00:58:02.439520 26457 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e195e456e2a4dfc953a21ab3f1dbeae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:02 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:34164
I20260806 00:58:02.439883 26924 tablet_replica.cc:330] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7: stopping tablet replica
I20260806 00:58:02.440006 26924 raft_consensus.cc:2229] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 1 LEADER]: Raft consensus shutting down.
I20260806 00:58:02.440138 26924 raft_consensus.cc:2258] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:02.440433 26924 ts_tablet_manager.cc:1905] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:02.442220 26925 tablet_replica.cc:330] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc: stopping tablet replica
I20260806 00:58:02.442457 26924 ts_tablet_manager.cc:1918] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 00:58:02.442529 26924 log.cc:1196] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-1/wal/wals/5e195e456e2a4dfc953a21ab3f1dbeae
I20260806 00:58:02.442847 26924 ts_tablet_manager.cc:1939] T 5e195e456e2a4dfc953a21ab3f1dbeae P 4650ad0a2c9f4d308af8035b8b5efeb7: Deleting consensus metadata
I20260806 00:58:02.443188 26925 raft_consensus.cc:2229] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:02.443265 26305 catalog_manager.cc:4841] TS 4650ad0a2c9f4d308af8035b8b5efeb7 (127.23.242.2:33789): tablet 5e195e456e2a4dfc953a21ab3f1dbeae (table test-table [id=88cd37fc0601404fb24222e9a1fc0208]) successfully deleted
I20260806 00:58:02.443344 26925 raft_consensus.cc:2258] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:02.443548 26926 tablet_replica.cc:330] T 5e195e456e2a4dfc953a21ab3f1dbeae P e57237ca962b4e81a69d191e8d9c1ab4: stopping tablet replica
I20260806 00:58:02.443629 26926 raft_consensus.cc:2229] T 5e195e456e2a4dfc953a21ab3f1dbeae P e57237ca962b4e81a69d191e8d9c1ab4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:02.443670 26925 ts_tablet_manager.cc:1905] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:02.443728 26926 raft_consensus.cc:2258] T 5e195e456e2a4dfc953a21ab3f1dbeae P e57237ca962b4e81a69d191e8d9c1ab4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:02.444005 26926 ts_tablet_manager.cc:1905] T 5e195e456e2a4dfc953a21ab3f1dbeae P e57237ca962b4e81a69d191e8d9c1ab4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:02.445552 26925 ts_tablet_manager.cc:1918] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 00:58:02.445631 26925 log.cc:1196] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-2/wal/wals/5e195e456e2a4dfc953a21ab3f1dbeae
I20260806 00:58:02.445897 26925 ts_tablet_manager.cc:1939] T 5e195e456e2a4dfc953a21ab3f1dbeae P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Deleting consensus metadata
I20260806 00:58:02.445902 26926 ts_tablet_manager.cc:1918] T 5e195e456e2a4dfc953a21ab3f1dbeae P e57237ca962b4e81a69d191e8d9c1ab4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 00:58:02.445959 26926 log.cc:1196] T 5e195e456e2a4dfc953a21ab3f1dbeae P e57237ca962b4e81a69d191e8d9c1ab4: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-0/wal/wals/5e195e456e2a4dfc953a21ab3f1dbeae
I20260806 00:58:02.446194 26926 ts_tablet_manager.cc:1939] T 5e195e456e2a4dfc953a21ab3f1dbeae P e57237ca962b4e81a69d191e8d9c1ab4: Deleting consensus metadata
I20260806 00:58:02.446339 26304 catalog_manager.cc:4841] TS 6e1948e9fd0b4766b3d8ac356d9d9fbc (127.23.242.3:39607): tablet 5e195e456e2a4dfc953a21ab3f1dbeae (table test-table [id=88cd37fc0601404fb24222e9a1fc0208]) successfully deleted
I20260806 00:58:02.446879 26302 catalog_manager.cc:4841] TS e57237ca962b4e81a69d191e8d9c1ab4 (127.23.242.1:36631): tablet 5e195e456e2a4dfc953a21ab3f1dbeae (table test-table [id=88cd37fc0601404fb24222e9a1fc0208]) successfully deleted
Aug 06 00:58:02 dist-test-slave-dhph krb5kdc[26273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977882, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 00:58:02 dist-test-slave-dhph krb5kdc[26273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977882, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:02.479308 26934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:02.464206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48826 (local address 127.23.242.62:41397)
0806 00:58:02.464427 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:02.464430 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:02.464497 (+ 67us) server_negotiation.cc:407] Connection header received
0806 00:58:02.464597 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:02.464598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:02.464611 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:02.464715 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:02.465652 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:02.466329 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:02.467821 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:02.468125 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:02.468224 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:02.475419 (+ 7195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:02.475421 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:02.475421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:02.475429 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:02.477139 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:02.477977 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:02.477978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:02.477979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:02.478048 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:02.478298 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:02.478299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:02.478300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:02.478429 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:02.478517 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:02.478981 (+ 464us) server_negotiation.cc:299] Negotiation successful
0806 00:58:02.479147 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260806 00:58:02.481896 26317 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 00:58:02.482141 26317 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 00:58:02.484392 26457 tablet_service.cc:1467] Processing CreateTablet for tablet f1cd47a6783b41f69dfe9f31898997a1 (DEFAULT_TABLE table=test-table [id=bcbc5ac834bb47dea9bb3eedc35b9c88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:02.484426 26744 tablet_service.cc:1467] Processing CreateTablet for tablet f1cd47a6783b41f69dfe9f31898997a1 (DEFAULT_TABLE table=test-table [id=bcbc5ac834bb47dea9bb3eedc35b9c88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:02.484405 26599 tablet_service.cc:1467] Processing CreateTablet for tablet f1cd47a6783b41f69dfe9f31898997a1 (DEFAULT_TABLE table=test-table [id=bcbc5ac834bb47dea9bb3eedc35b9c88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:02.484589 26744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1cd47a6783b41f69dfe9f31898997a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:02.484589 26599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1cd47a6783b41f69dfe9f31898997a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:02.484640 26457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1cd47a6783b41f69dfe9f31898997a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:02.486112 26935 tablet_bootstrap.cc:492] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7: Bootstrap starting.
I20260806 00:58:02.486222 26937 tablet_bootstrap.cc:492] T f1cd47a6783b41f69dfe9f31898997a1 P e57237ca962b4e81a69d191e8d9c1ab4: Bootstrap starting.
I20260806 00:58:02.486347 26936 tablet_bootstrap.cc:492] T f1cd47a6783b41f69dfe9f31898997a1 P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Bootstrap starting.
I20260806 00:58:02.486620 26935 tablet_bootstrap.cc:654] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:02.486670 26937 tablet_bootstrap.cc:654] T f1cd47a6783b41f69dfe9f31898997a1 P e57237ca962b4e81a69d191e8d9c1ab4: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:02.486711 26936 tablet_bootstrap.cc:654] T f1cd47a6783b41f69dfe9f31898997a1 P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:02.487347 26937 tablet_bootstrap.cc:492] T f1cd47a6783b41f69dfe9f31898997a1 P e57237ca962b4e81a69d191e8d9c1ab4: No bootstrap required, opened a new log
I20260806 00:58:02.487394 26937 ts_tablet_manager.cc:1397] T f1cd47a6783b41f69dfe9f31898997a1 P e57237ca962b4e81a69d191e8d9c1ab4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:02.487429 26935 tablet_bootstrap.cc:492] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7: No bootstrap required, opened a new log
I20260806 00:58:02.487484 26935 ts_tablet_manager.cc:1397] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:02.487506 26937 raft_consensus.cc:348] T f1cd47a6783b41f69dfe9f31898997a1 P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:02.487586 26937 raft_consensus.cc:374] T f1cd47a6783b41f69dfe9f31898997a1 P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:02.487617 26937 raft_consensus.cc:729] T f1cd47a6783b41f69dfe9f31898997a1 P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e57237ca962b4e81a69d191e8d9c1ab4, State: Initialized, Role: FOLLOWER
I20260806 00:58:02.487609 26935 raft_consensus.cc:348] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:02.487668 26937 consensus_queue.cc:260] T f1cd47a6783b41f69dfe9f31898997a1 P e57237ca962b4e81a69d191e8d9c1ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:02.487692 26935 raft_consensus.cc:374] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:02.487720 26935 raft_consensus.cc:729] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4650ad0a2c9f4d308af8035b8b5efeb7, State: Initialized, Role: FOLLOWER
I20260806 00:58:02.487767 26937 ts_tablet_manager.cc:1428] T f1cd47a6783b41f69dfe9f31898997a1 P e57237ca962b4e81a69d191e8d9c1ab4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:02.487782 26935 consensus_queue.cc:260] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:02.487890 26935 ts_tablet_manager.cc:1428] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:02.488379 26936 tablet_bootstrap.cc:492] T f1cd47a6783b41f69dfe9f31898997a1 P 6e1948e9fd0b4766b3d8ac356d9d9fbc: No bootstrap required, opened a new log
I20260806 00:58:02.488471 26936 ts_tablet_manager.cc:1397] T f1cd47a6783b41f69dfe9f31898997a1 P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 00:58:02.488653 26936 raft_consensus.cc:348] T f1cd47a6783b41f69dfe9f31898997a1 P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:02.488739 26936 raft_consensus.cc:374] T f1cd47a6783b41f69dfe9f31898997a1 P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:02.488766 26936 raft_consensus.cc:729] T f1cd47a6783b41f69dfe9f31898997a1 P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e1948e9fd0b4766b3d8ac356d9d9fbc, State: Initialized, Role: FOLLOWER
I20260806 00:58:02.488822 26936 consensus_queue.cc:260] T f1cd47a6783b41f69dfe9f31898997a1 P 6e1948e9fd0b4766b3d8ac356d9d9fbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:02.488998 26936 ts_tablet_manager.cc:1428] T f1cd47a6783b41f69dfe9f31898997a1 P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260806 00:58:02.570262 26918 raft_consensus.cc:482] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 00:58:02.570398 26918 raft_consensus.cc:504] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:02.570706 26918 leader_election.cc:288] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e57237ca962b4e81a69d191e8d9c1ab4 (127.23.242.1:36631), 6e1948e9fd0b4766b3d8ac356d9d9fbc (127.23.242.3:39607)
I20260806 00:58:02.570955 26478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1cd47a6783b41f69dfe9f31898997a1" candidate_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" is_pre_election: true
I20260806 00:58:02.571092 26478 raft_consensus.cc:2454] T f1cd47a6783b41f69dfe9f31898997a1 P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4650ad0a2c9f4d308af8035b8b5efeb7 in term 0.
I20260806 00:58:02.571193 26763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1cd47a6783b41f69dfe9f31898997a1" candidate_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" is_pre_election: true
I20260806 00:58:02.571313 26763 raft_consensus.cc:2454] T f1cd47a6783b41f69dfe9f31898997a1 P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4650ad0a2c9f4d308af8035b8b5efeb7 in term 0.
I20260806 00:58:02.571321 26550 leader_election.cc:302] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4650ad0a2c9f4d308af8035b8b5efeb7, e57237ca962b4e81a69d191e8d9c1ab4; no voters:
I20260806 00:58:02.571409 26918 raft_consensus.cc:2781] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 00:58:02.571444 26918 raft_consensus.cc:482] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 00:58:02.571468 26918 raft_consensus.cc:3037] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:02.572288 26918 raft_consensus.cc:504] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:02.572448 26918 leader_election.cc:288] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7 [CANDIDATE]: Term 1 election: Requested vote from peers e57237ca962b4e81a69d191e8d9c1ab4 (127.23.242.1:36631), 6e1948e9fd0b4766b3d8ac356d9d9fbc (127.23.242.3:39607)
I20260806 00:58:02.572623 26763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1cd47a6783b41f69dfe9f31898997a1" candidate_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc"
I20260806 00:58:02.572733 26763 raft_consensus.cc:3037] T f1cd47a6783b41f69dfe9f31898997a1 P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:02.572707 26478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1cd47a6783b41f69dfe9f31898997a1" candidate_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e57237ca962b4e81a69d191e8d9c1ab4"
I20260806 00:58:02.572785 26478 raft_consensus.cc:3037] T f1cd47a6783b41f69dfe9f31898997a1 P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:02.573558 26478 raft_consensus.cc:2454] T f1cd47a6783b41f69dfe9f31898997a1 P e57237ca962b4e81a69d191e8d9c1ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4650ad0a2c9f4d308af8035b8b5efeb7 in term 1.
I20260806 00:58:02.573558 26763 raft_consensus.cc:2454] T f1cd47a6783b41f69dfe9f31898997a1 P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4650ad0a2c9f4d308af8035b8b5efeb7 in term 1.
I20260806 00:58:02.573777 26550 leader_election.cc:302] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4650ad0a2c9f4d308af8035b8b5efeb7, e57237ca962b4e81a69d191e8d9c1ab4; no voters:
I20260806 00:58:02.573863 26918 raft_consensus.cc:2781] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:02.573906 26918 raft_consensus.cc:686] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 1 LEADER]: Becoming Leader. State: Replica: 4650ad0a2c9f4d308af8035b8b5efeb7, State: Running, Role: LEADER
I20260806 00:58:02.573956 26918 consensus_queue.cc:237] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:02.574703 26317 catalog_manager.cc:5495] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4650ad0a2c9f4d308af8035b8b5efeb7 (127.23.242.2). New cstate: current_term: 1 leader_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } health_report { overall_health: UNKNOWN } } }
I20260806 00:58:02.589536 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:02.585717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45422 (local address 127.23.242.2:33789)
0806 00:58:02.585888 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:02.585890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:02.585896 (+ 6us) server_negotiation.cc:407] Connection header received
0806 00:58:02.585914 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:02.585914 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:02.585924 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:02.585976 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 00:58:02.586139 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:02.587255 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:02.588473 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:02.588749 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:02.588818 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:02.589227 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 00:58:02.589264 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 00:58:02.589332 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:02.589398 (+ 66us) server_negotiation.cc:299] Negotiation successful
0806 00:58:02.589426 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":110,"threads_started":1}
I20260806 00:58:02.593365 26478 raft_consensus.cc:1261] T f1cd47a6783b41f69dfe9f31898997a1 P e57237ca962b4e81a69d191e8d9c1ab4 [term 1 FOLLOWER]: Refusing update from remote peer 4650ad0a2c9f4d308af8035b8b5efeb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:02.593560 26764 raft_consensus.cc:1261] T f1cd47a6783b41f69dfe9f31898997a1 P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 1 FOLLOWER]: Refusing update from remote peer 4650ad0a2c9f4d308af8035b8b5efeb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:02.593822 26900 consensus_queue.cc:1035] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:02.593952 26900 consensus_queue.cc:1035] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:02.614867 26315 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:58:02.614997 26315 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:58:02.615891 26315 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70823b4c6d02429ca702661edb1041e8: Sending DeleteTablet for 3 replicas of tablet f1cd47a6783b41f69dfe9f31898997a1
I20260806 00:58:02.616182 26457 tablet_service.cc:1514] Processing DeleteTablet for tablet f1cd47a6783b41f69dfe9f31898997a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:02 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:34164
I20260806 00:58:02.616293 26926 tablet_replica.cc:330] T f1cd47a6783b41f69dfe9f31898997a1 P e57237ca962b4e81a69d191e8d9c1ab4: stopping tablet replica
I20260806 00:58:02.616274 26744 tablet_service.cc:1514] Processing DeleteTablet for tablet f1cd47a6783b41f69dfe9f31898997a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:02 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:39620
I20260806 00:58:02.616307 26599 tablet_service.cc:1514] Processing DeleteTablet for tablet f1cd47a6783b41f69dfe9f31898997a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:02 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:45398
I20260806 00:58:02.616353 26926 raft_consensus.cc:2229] T f1cd47a6783b41f69dfe9f31898997a1 P e57237ca962b4e81a69d191e8d9c1ab4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:02.616366 26925 tablet_replica.cc:330] T f1cd47a6783b41f69dfe9f31898997a1 P 6e1948e9fd0b4766b3d8ac356d9d9fbc: stopping tablet replica
I20260806 00:58:02.616400 26926 raft_consensus.cc:2258] T f1cd47a6783b41f69dfe9f31898997a1 P e57237ca962b4e81a69d191e8d9c1ab4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:02.616408 26925 raft_consensus.cc:2229] T f1cd47a6783b41f69dfe9f31898997a1 P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:02.616429 26924 tablet_replica.cc:330] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7: stopping tablet replica
I20260806 00:58:02.616454 26925 raft_consensus.cc:2258] T f1cd47a6783b41f69dfe9f31898997a1 P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:02.616490 26924 raft_consensus.cc:2229] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 1 LEADER]: Raft consensus shutting down.
I20260806 00:58:02.616564 26924 raft_consensus.cc:2258] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:02.616683 26315 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 00:58:02.616852 26315 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 00:58:02.616902 26925 ts_tablet_manager.cc:1905] T f1cd47a6783b41f69dfe9f31898997a1 P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:02.617303 26926 ts_tablet_manager.cc:1905] T f1cd47a6783b41f69dfe9f31898997a1 P e57237ca962b4e81a69d191e8d9c1ab4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:02.619033 26924 ts_tablet_manager.cc:1905] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:02.619519 26925 ts_tablet_manager.cc:1918] T f1cd47a6783b41f69dfe9f31898997a1 P 6e1948e9fd0b4766b3d8ac356d9d9fbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 00:58:02.619568 26926 ts_tablet_manager.cc:1918] T f1cd47a6783b41f69dfe9f31898997a1 P e57237ca962b4e81a69d191e8d9c1ab4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 00:58:02.619591 26925 log.cc:1196] T f1cd47a6783b41f69dfe9f31898997a1 P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-2/wal/wals/f1cd47a6783b41f69dfe9f31898997a1
I20260806 00:58:02.619624 26926 log.cc:1196] T f1cd47a6783b41f69dfe9f31898997a1 P e57237ca962b4e81a69d191e8d9c1ab4: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-0/wal/wals/f1cd47a6783b41f69dfe9f31898997a1
I20260806 00:58:02.619861 26926 ts_tablet_manager.cc:1939] T f1cd47a6783b41f69dfe9f31898997a1 P e57237ca962b4e81a69d191e8d9c1ab4: Deleting consensus metadata
I20260806 00:58:02.619886 26925 ts_tablet_manager.cc:1939] T f1cd47a6783b41f69dfe9f31898997a1 P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Deleting consensus metadata
I20260806 00:58:02.620182 26302 catalog_manager.cc:4841] TS e57237ca962b4e81a69d191e8d9c1ab4 (127.23.242.1:36631): tablet f1cd47a6783b41f69dfe9f31898997a1 (table test-table [id=bcbc5ac834bb47dea9bb3eedc35b9c88]) successfully deleted
I20260806 00:58:02.620270 26457 tablet_service.cc:1467] Processing CreateTablet for tablet 9d065e6eb44f4c858bde677e472ec0d5 (DEFAULT_TABLE table=test-table [id=3319bb8afa8f41b4a5f275ea290dbc0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:02.620306 26744 tablet_service.cc:1467] Processing CreateTablet for tablet 9d065e6eb44f4c858bde677e472ec0d5 (DEFAULT_TABLE table=test-table [id=3319bb8afa8f41b4a5f275ea290dbc0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:02.620402 26744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d065e6eb44f4c858bde677e472ec0d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:02.620402 26457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d065e6eb44f4c858bde677e472ec0d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:02.621129 26924 ts_tablet_manager.cc:1918] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 00:58:02.621227 26924 log.cc:1196] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-1/wal/wals/f1cd47a6783b41f69dfe9f31898997a1
I20260806 00:58:02.621434 26924 ts_tablet_manager.cc:1939] T f1cd47a6783b41f69dfe9f31898997a1 P 4650ad0a2c9f4d308af8035b8b5efeb7: Deleting consensus metadata
I20260806 00:58:02.621798 26936 tablet_bootstrap.cc:492] T 9d065e6eb44f4c858bde677e472ec0d5 P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Bootstrap starting.
I20260806 00:58:02.621879 26937 tablet_bootstrap.cc:492] T 9d065e6eb44f4c858bde677e472ec0d5 P e57237ca962b4e81a69d191e8d9c1ab4: Bootstrap starting.
I20260806 00:58:02.622114 26599 tablet_service.cc:1467] Processing CreateTablet for tablet 9d065e6eb44f4c858bde677e472ec0d5 (DEFAULT_TABLE table=test-table [id=3319bb8afa8f41b4a5f275ea290dbc0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:02.622238 26599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d065e6eb44f4c858bde677e472ec0d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:02.622246 26937 tablet_bootstrap.cc:654] T 9d065e6eb44f4c858bde677e472ec0d5 P e57237ca962b4e81a69d191e8d9c1ab4: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:02.622290 26936 tablet_bootstrap.cc:654] T 9d065e6eb44f4c858bde677e472ec0d5 P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:02.622948 26936 tablet_bootstrap.cc:492] T 9d065e6eb44f4c858bde677e472ec0d5 P 6e1948e9fd0b4766b3d8ac356d9d9fbc: No bootstrap required, opened a new log
I20260806 00:58:02.622937 26305 catalog_manager.cc:4841] TS 4650ad0a2c9f4d308af8035b8b5efeb7 (127.23.242.2:33789): tablet f1cd47a6783b41f69dfe9f31898997a1 (table test-table [id=bcbc5ac834bb47dea9bb3eedc35b9c88]) successfully deleted
I20260806 00:58:02.623006 26936 ts_tablet_manager.cc:1397] T 9d065e6eb44f4c858bde677e472ec0d5 P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:02.623142 26937 tablet_bootstrap.cc:492] T 9d065e6eb44f4c858bde677e472ec0d5 P e57237ca962b4e81a69d191e8d9c1ab4: No bootstrap required, opened a new log
I20260806 00:58:02.623183 26937 ts_tablet_manager.cc:1397] T 9d065e6eb44f4c858bde677e472ec0d5 P e57237ca962b4e81a69d191e8d9c1ab4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:02.623180 26936 raft_consensus.cc:348] T 9d065e6eb44f4c858bde677e472ec0d5 P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:02.623251 26936 raft_consensus.cc:374] T 9d065e6eb44f4c858bde677e472ec0d5 P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:02.623271 26936 raft_consensus.cc:729] T 9d065e6eb44f4c858bde677e472ec0d5 P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e1948e9fd0b4766b3d8ac356d9d9fbc, State: Initialized, Role: FOLLOWER
I20260806 00:58:02.623289 26937 raft_consensus.cc:348] T 9d065e6eb44f4c858bde677e472ec0d5 P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:02.623317 26936 consensus_queue.cc:260] T 9d065e6eb44f4c858bde677e472ec0d5 P 6e1948e9fd0b4766b3d8ac356d9d9fbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:02.623351 26937 raft_consensus.cc:374] T 9d065e6eb44f4c858bde677e472ec0d5 P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:02.623376 26937 raft_consensus.cc:729] T 9d065e6eb44f4c858bde677e472ec0d5 P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e57237ca962b4e81a69d191e8d9c1ab4, State: Initialized, Role: FOLLOWER
I20260806 00:58:02.623409 26936 ts_tablet_manager.cc:1428] T 9d065e6eb44f4c858bde677e472ec0d5 P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:02.623425 26937 consensus_queue.cc:260] T 9d065e6eb44f4c858bde677e472ec0d5 P e57237ca962b4e81a69d191e8d9c1ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:02.623530 26937 ts_tablet_manager.cc:1428] T 9d065e6eb44f4c858bde677e472ec0d5 P e57237ca962b4e81a69d191e8d9c1ab4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:02.623819 26935 tablet_bootstrap.cc:492] T 9d065e6eb44f4c858bde677e472ec0d5 P 4650ad0a2c9f4d308af8035b8b5efeb7: Bootstrap starting.
I20260806 00:58:02.624150 26304 catalog_manager.cc:4841] TS 6e1948e9fd0b4766b3d8ac356d9d9fbc (127.23.242.3:39607): tablet f1cd47a6783b41f69dfe9f31898997a1 (table test-table [id=bcbc5ac834bb47dea9bb3eedc35b9c88]) successfully deleted
I20260806 00:58:02.624187 26935 tablet_bootstrap.cc:654] T 9d065e6eb44f4c858bde677e472ec0d5 P 4650ad0a2c9f4d308af8035b8b5efeb7: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:02.624768 26935 tablet_bootstrap.cc:492] T 9d065e6eb44f4c858bde677e472ec0d5 P 4650ad0a2c9f4d308af8035b8b5efeb7: No bootstrap required, opened a new log
I20260806 00:58:02.624817 26935 ts_tablet_manager.cc:1397] T 9d065e6eb44f4c858bde677e472ec0d5 P 4650ad0a2c9f4d308af8035b8b5efeb7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:02.624938 26935 raft_consensus.cc:348] T 9d065e6eb44f4c858bde677e472ec0d5 P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:02.625006 26935 raft_consensus.cc:374] T 9d065e6eb44f4c858bde677e472ec0d5 P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:02.625021 26935 raft_consensus.cc:729] T 9d065e6eb44f4c858bde677e472ec0d5 P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4650ad0a2c9f4d308af8035b8b5efeb7, State: Initialized, Role: FOLLOWER
I20260806 00:58:02.625077 26935 consensus_queue.cc:260] T 9d065e6eb44f4c858bde677e472ec0d5 P 4650ad0a2c9f4d308af8035b8b5efeb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:02.625174 26935 ts_tablet_manager.cc:1428] T 9d065e6eb44f4c858bde677e472ec0d5 P 4650ad0a2c9f4d308af8035b8b5efeb7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260806 00:58:02.923130 26843 raft_consensus.cc:482] T 9d065e6eb44f4c858bde677e472ec0d5 P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 00:58:02.923270 26843 raft_consensus.cc:504] T 9d065e6eb44f4c858bde677e472ec0d5 P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:02.923480 26843 leader_election.cc:288] T 9d065e6eb44f4c858bde677e472ec0d5 P e57237ca962b4e81a69d191e8d9c1ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4650ad0a2c9f4d308af8035b8b5efeb7 (127.23.242.2:33789), 6e1948e9fd0b4766b3d8ac356d9d9fbc (127.23.242.3:39607)
I20260806 00:58:02.923834 26763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d065e6eb44f4c858bde677e472ec0d5" candidate_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" is_pre_election: true
I20260806 00:58:02.923934 26763 raft_consensus.cc:2454] T 9d065e6eb44f4c858bde677e472ec0d5 P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e57237ca962b4e81a69d191e8d9c1ab4 in term 0.
I20260806 00:58:02.923879 26619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d065e6eb44f4c858bde677e472ec0d5" candidate_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" is_pre_election: true
I20260806 00:58:02.924058 26619 raft_consensus.cc:2454] T 9d065e6eb44f4c858bde677e472ec0d5 P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e57237ca962b4e81a69d191e8d9c1ab4 in term 0.
I20260806 00:58:02.924180 26410 leader_election.cc:302] T 9d065e6eb44f4c858bde677e472ec0d5 P e57237ca962b4e81a69d191e8d9c1ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e1948e9fd0b4766b3d8ac356d9d9fbc, e57237ca962b4e81a69d191e8d9c1ab4; no voters:
I20260806 00:58:02.924309 26843 raft_consensus.cc:2781] T 9d065e6eb44f4c858bde677e472ec0d5 P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 00:58:02.924356 26843 raft_consensus.cc:482] T 9d065e6eb44f4c858bde677e472ec0d5 P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 00:58:02.924373 26843 raft_consensus.cc:3037] T 9d065e6eb44f4c858bde677e472ec0d5 P e57237ca962b4e81a69d191e8d9c1ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:02.925079 26843 raft_consensus.cc:504] T 9d065e6eb44f4c858bde677e472ec0d5 P e57237ca962b4e81a69d191e8d9c1ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:02.925241 26843 leader_election.cc:288] T 9d065e6eb44f4c858bde677e472ec0d5 P e57237ca962b4e81a69d191e8d9c1ab4 [CANDIDATE]: Term 1 election: Requested vote from peers 4650ad0a2c9f4d308af8035b8b5efeb7 (127.23.242.2:33789), 6e1948e9fd0b4766b3d8ac356d9d9fbc (127.23.242.3:39607)
I20260806 00:58:02.925408 26763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d065e6eb44f4c858bde677e472ec0d5" candidate_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc"
I20260806 00:58:02.925419 26619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d065e6eb44f4c858bde677e472ec0d5" candidate_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7"
I20260806 00:58:02.925508 26619 raft_consensus.cc:3037] T 9d065e6eb44f4c858bde677e472ec0d5 P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:02.925510 26763 raft_consensus.cc:3037] T 9d065e6eb44f4c858bde677e472ec0d5 P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:02.926151 26763 raft_consensus.cc:2454] T 9d065e6eb44f4c858bde677e472ec0d5 P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e57237ca962b4e81a69d191e8d9c1ab4 in term 1.
I20260806 00:58:02.926343 26619 raft_consensus.cc:2454] T 9d065e6eb44f4c858bde677e472ec0d5 P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e57237ca962b4e81a69d191e8d9c1ab4 in term 1.
I20260806 00:58:02.926385 26410 leader_election.cc:302] T 9d065e6eb44f4c858bde677e472ec0d5 P e57237ca962b4e81a69d191e8d9c1ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e1948e9fd0b4766b3d8ac356d9d9fbc, e57237ca962b4e81a69d191e8d9c1ab4; no voters:
I20260806 00:58:02.926472 26843 raft_consensus.cc:2781] T 9d065e6eb44f4c858bde677e472ec0d5 P e57237ca962b4e81a69d191e8d9c1ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:02.926556 26843 raft_consensus.cc:686] T 9d065e6eb44f4c858bde677e472ec0d5 P e57237ca962b4e81a69d191e8d9c1ab4 [term 1 LEADER]: Becoming Leader. State: Replica: e57237ca962b4e81a69d191e8d9c1ab4, State: Running, Role: LEADER
I20260806 00:58:02.926645 26843 consensus_queue.cc:237] T 9d065e6eb44f4c858bde677e472ec0d5 P e57237ca962b4e81a69d191e8d9c1ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } }
I20260806 00:58:02.927449 26315 catalog_manager.cc:5495] T 9d065e6eb44f4c858bde677e472ec0d5 P e57237ca962b4e81a69d191e8d9c1ab4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e57237ca962b4e81a69d191e8d9c1ab4 (127.23.242.1). New cstate: current_term: 1 leader_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e57237ca962b4e81a69d191e8d9c1ab4" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 } health_report { overall_health: UNKNOWN } } }
I20260806 00:58:02.988601 26952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:02.985530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34202 (local address 127.23.242.1:36631)
0806 00:58:02.985911 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:02.985913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:02.985922 (+ 9us) server_negotiation.cc:407] Connection header received
0806 00:58:02.985954 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:02.985955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:02.985967 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:02.986059 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 00:58:02.986203 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:02.986906 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:02.987802 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:02.988039 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:02.988125 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:02.988296 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 00:58:02.988339 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 00:58:02.988406 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:02.988466 (+ 60us) server_negotiation.cc:299] Negotiation successful
0806 00:58:02.988491 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":272,"threads_started":1}
I20260806 00:58:02.991487 26763 raft_consensus.cc:1261] T 9d065e6eb44f4c858bde677e472ec0d5 P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 1 FOLLOWER]: Refusing update from remote peer e57237ca962b4e81a69d191e8d9c1ab4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:02.991602 26619 raft_consensus.cc:1261] T 9d065e6eb44f4c858bde677e472ec0d5 P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 1 FOLLOWER]: Refusing update from remote peer e57237ca962b4e81a69d191e8d9c1ab4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:02.991837 26919 consensus_queue.cc:1035] T 9d065e6eb44f4c858bde677e472ec0d5 P e57237ca962b4e81a69d191e8d9c1ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4650ad0a2c9f4d308af8035b8b5efeb7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 33789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:02.992055 26843 consensus_queue.cc:1035] T 9d065e6eb44f4c858bde677e472ec0d5 P e57237ca962b4e81a69d191e8d9c1ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e1948e9fd0b4766b3d8ac356d9d9fbc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 00:58:02.998868 26903 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 00:58:04.528813 26318 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:58:04.528959 26318 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:58:04.530404 26318 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70823b4c6d02429ca702661edb1041e8: Sending DeleteTablet for 3 replicas of tablet 9d065e6eb44f4c858bde677e472ec0d5
I20260806 00:58:04.530815 26456 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d065e6eb44f4c858bde677e472ec0d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:04 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:34164
I20260806 00:58:04.530812 26599 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d065e6eb44f4c858bde677e472ec0d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:04 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:45398
I20260806 00:58:04.530942 26744 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d065e6eb44f4c858bde677e472ec0d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:04 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:39620
I20260806 00:58:04.531066 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 26392
I20260806 00:58:04.535203 26978 tablet_replica.cc:330] T 9d065e6eb44f4c858bde677e472ec0d5 P 4650ad0a2c9f4d308af8035b8b5efeb7: stopping tablet replica
I20260806 00:58:04.535939 26978 raft_consensus.cc:2229] T 9d065e6eb44f4c858bde677e472ec0d5 P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:04.536034 26978 raft_consensus.cc:2258] T 9d065e6eb44f4c858bde677e472ec0d5 P 4650ad0a2c9f4d308af8035b8b5efeb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:04.536659 26981 tablet_replica.cc:330] T 9d065e6eb44f4c858bde677e472ec0d5 P 6e1948e9fd0b4766b3d8ac356d9d9fbc: stopping tablet replica
I20260806 00:58:04.536737 26981 raft_consensus.cc:2229] T 9d065e6eb44f4c858bde677e472ec0d5 P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:04.536787 26981 raft_consensus.cc:2258] T 9d065e6eb44f4c858bde677e472ec0d5 P 6e1948e9fd0b4766b3d8ac356d9d9fbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:04.537102 26981 ts_tablet_manager.cc:1905] T 9d065e6eb44f4c858bde677e472ec0d5 P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:04.537263 26978 ts_tablet_manager.cc:1905] T 9d065e6eb44f4c858bde677e472ec0d5 P 4650ad0a2c9f4d308af8035b8b5efeb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:04.539963 26981 ts_tablet_manager.cc:1918] T 9d065e6eb44f4c858bde677e472ec0d5 P 6e1948e9fd0b4766b3d8ac356d9d9fbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 00:58:04.540074 26981 log.cc:1196] T 9d065e6eb44f4c858bde677e472ec0d5 P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-2/wal/wals/9d065e6eb44f4c858bde677e472ec0d5
I20260806 00:58:04.540535 26981 ts_tablet_manager.cc:1939] T 9d065e6eb44f4c858bde677e472ec0d5 P 6e1948e9fd0b4766b3d8ac356d9d9fbc: Deleting consensus metadata
I20260806 00:58:04.540848 26978 ts_tablet_manager.cc:1918] T 9d065e6eb44f4c858bde677e472ec0d5 P 4650ad0a2c9f4d308af8035b8b5efeb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 00:58:04.541366 26304 catalog_manager.cc:4841] TS 6e1948e9fd0b4766b3d8ac356d9d9fbc (127.23.242.3:39607): tablet 9d065e6eb44f4c858bde677e472ec0d5 (table test-table [id=3319bb8afa8f41b4a5f275ea290dbc0d]) successfully deleted
I20260806 00:58:04.541638 26978 log.cc:1196] T 9d065e6eb44f4c858bde677e472ec0d5 P 4650ad0a2c9f4d308af8035b8b5efeb7: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785977869987768-24520-0/minicluster-data/ts-1/wal/wals/9d065e6eb44f4c858bde677e472ec0d5
I20260806 00:58:04.541939 26978 ts_tablet_manager.cc:1939] T 9d065e6eb44f4c858bde677e472ec0d5 P 4650ad0a2c9f4d308af8035b8b5efeb7: Deleting consensus metadata
I20260806 00:58:04.542495 26305 catalog_manager.cc:4841] TS 4650ad0a2c9f4d308af8035b8b5efeb7 (127.23.242.2:33789): tablet 9d065e6eb44f4c858bde677e472ec0d5 (table test-table [id=3319bb8afa8f41b4a5f275ea290dbc0d]) successfully deleted
W20260806 00:58:04.544327 26302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.242.1:36631 (error 108)
I20260806 00:58:04.545079 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 26534
I20260806 00:58:04.545209 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:04.544916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:36631 (local address 127.0.0.1:34210)
0806 00:58:04.545076 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:04.545150 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.1:36631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":108,"threads_started":1}
W20260806 00:58:04.546823 26302 catalog_manager.cc:4584] TS e57237ca962b4e81a69d191e8d9c1ab4 (127.23.242.1:36631): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9d065e6eb44f4c858bde677e472ec0d5: Network error: Client connection negotiation failed: client connection to 127.23.242.1:36631: connect: Connection refused (error 111)
I20260806 00:58:04.553143 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 26675
I20260806 00:58:04.560624 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 26286
2026-08-06T00:58:04Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6444 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 00:58:04 dist-test-slave-dhph krb5kdc[26986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 00:58:04 dist-test-slave-dhph krb5kdc[26986](info): set up 2 sockets
Aug 06 00:58:04 dist-test-slave-dhph krb5kdc[26986](info): commencing operation
krb5kdc: starting...
W20260806 00:58:05.822465 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 00:58:05 dist-test-slave-dhph krb5kdc[26986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T00:58:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T00:58:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 00:58:05.978171 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:39243
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:39243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:06.076973 27000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:06.077554 27000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:06.077736 27000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:06.080513 27000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 00:58:06.080632 27000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:06.080670 27000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:06.080710 27000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 00:58:06.080742 27000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 00:58:06.083201 27000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:39243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:39243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:06.083788 27000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:06.084103 27000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:06.089972 27005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:06.090044 27006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:06.089982 27008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:06.090819 27000 server_base.cc:1034] running on GCE node
I20260806 00:58:06.091123 27000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:06.091460 27000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:06.092713 27000 hybrid_clock.cc:648] HybridClock initialized: now 1785977886092683 us; error 46 us; skew 500 ppm
Aug 06 00:58:06 dist-test-slave-dhph krb5kdc[26986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977886, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:06.097635 27000 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 00:58:06.098831 27000 webserver.cc:460] Webserver started at http://127.23.242.62:37855/ using document root <none> and password file <none>
I20260806 00:58:06.099223 27000 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:06.099313 27000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:06.099514 27000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:06.100955 27000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "fe15af5ea2554c54ba97ba61059aa957"
format_stamp: "Formatted at 2026-08-06 00:58:06 on dist-test-slave-dhph"
server_key: "b7bb49ffb7401b03ef29dad78cb9c8e6"
server_key_iv: "ca440a93906dde2c51fec8132c936671"
server_key_version: "encryptionkey@0"
I20260806 00:58:06.101416 27000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "fe15af5ea2554c54ba97ba61059aa957"
format_stamp: "Formatted at 2026-08-06 00:58:06 on dist-test-slave-dhph"
server_key: "b7bb49ffb7401b03ef29dad78cb9c8e6"
server_key_iv: "ca440a93906dde2c51fec8132c936671"
server_key_version: "encryptionkey@0"
I20260806 00:58:06.103737 27000 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.005s sys 0.000s
I20260806 00:58:06.104900 27015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:06.105225 27000 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:06.105461 27000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "fe15af5ea2554c54ba97ba61059aa957"
format_stamp: "Formatted at 2026-08-06 00:58:06 on dist-test-slave-dhph"
server_key: "b7bb49ffb7401b03ef29dad78cb9c8e6"
server_key_iv: "ca440a93906dde2c51fec8132c936671"
server_key_version: "encryptionkey@0"
I20260806 00:58:06.105569 27000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:06.126529 27000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:06.129383 27000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:06.129639 27000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:06.138111 27000 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:39243
I20260806 00:58:06.138237 27067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:39243 every 8 connection(s)
I20260806 00:58:06.138695 27000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 00:58:06.139946 27068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:06.143193 27068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe15af5ea2554c54ba97ba61059aa957: Bootstrap starting.
I20260806 00:58:06.144282 27068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe15af5ea2554c54ba97ba61059aa957: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:06.144695 27068 log.cc:824] T 00000000000000000000000000000000 P fe15af5ea2554c54ba97ba61059aa957: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:06.145543 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 27000
I20260806 00:58:06.145639 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 00:58:06.145785 27068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe15af5ea2554c54ba97ba61059aa957: No bootstrap required, opened a new log
I20260806 00:58:06.145828 24520 external_mini_cluster.cc:1428] Setting key 9d9163d59d6a3129c503f0fda693e2cc
I20260806 00:58:06.148231 27068 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe15af5ea2554c54ba97ba61059aa957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe15af5ea2554c54ba97ba61059aa957" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39243 } }
I20260806 00:58:06.148438 27068 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe15af5ea2554c54ba97ba61059aa957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:06.148499 27068 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe15af5ea2554c54ba97ba61059aa957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe15af5ea2554c54ba97ba61059aa957, State: Initialized, Role: FOLLOWER
I20260806 00:58:06.148756 27068 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe15af5ea2554c54ba97ba61059aa957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe15af5ea2554c54ba97ba61059aa957" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39243 } }
I20260806 00:58:06.148882 27068 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe15af5ea2554c54ba97ba61059aa957 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 00:58:06.148936 27068 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe15af5ea2554c54ba97ba61059aa957 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 00:58:06.149115 27068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe15af5ea2554c54ba97ba61059aa957 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:06.151085 27068 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe15af5ea2554c54ba97ba61059aa957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe15af5ea2554c54ba97ba61059aa957" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39243 } }
I20260806 00:58:06.151364 27068 leader_election.cc:302] T 00000000000000000000000000000000 P fe15af5ea2554c54ba97ba61059aa957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe15af5ea2554c54ba97ba61059aa957; no voters:
I20260806 00:58:06.151646 27068 leader_election.cc:288] T 00000000000000000000000000000000 P fe15af5ea2554c54ba97ba61059aa957 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 00:58:06.151726 27073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe15af5ea2554c54ba97ba61059aa957 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:06.151891 27073 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe15af5ea2554c54ba97ba61059aa957 [term 1 LEADER]: Becoming Leader. State: Replica: fe15af5ea2554c54ba97ba61059aa957, State: Running, Role: LEADER
I20260806 00:58:06.151986 27068 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe15af5ea2554c54ba97ba61059aa957 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 00:58:06.152068 27073 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe15af5ea2554c54ba97ba61059aa957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe15af5ea2554c54ba97ba61059aa957" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39243 } }
I20260806 00:58:06.152513 27073 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe15af5ea2554c54ba97ba61059aa957 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe15af5ea2554c54ba97ba61059aa957. Latest consensus state: current_term: 1 leader_uuid: "fe15af5ea2554c54ba97ba61059aa957" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe15af5ea2554c54ba97ba61059aa957" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39243 } } }
I20260806 00:58:06.152519 27074 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe15af5ea2554c54ba97ba61059aa957 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe15af5ea2554c54ba97ba61059aa957" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe15af5ea2554c54ba97ba61059aa957" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39243 } } }
I20260806 00:58:06.152676 27073 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe15af5ea2554c54ba97ba61059aa957 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:06.152722 27074 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe15af5ea2554c54ba97ba61059aa957 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:06.153092 27081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 00:58:06.154381 27081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 06 00:58:06 dist-test-slave-dhph krb5kdc[26986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:06.157706 27081 catalog_manager.cc:1347] Generated new cluster ID: 43ac6fe5b69b49acbfe6c0d0bb44414a
I20260806 00:58:06.157917 27081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 00:58:06.171725 27081 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 00:58:06.171981 27072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:06.146485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36928 (local address 127.23.242.62:39243)
0806 00:58:06.146737 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:06.146747 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:06.146784 (+ 37us) server_negotiation.cc:407] Connection header received
0806 00:58:06.148287 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:06.148302 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:06.148411 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:06.148702 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:06.150208 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:06.151647 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:06.152852 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:06.153398 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:06.153528 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:06.167364 (+ 13836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:06.167369 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:06.167370 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:06.167387 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:06.169889 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:06.170471 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:06.170474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:06.170475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:06.170635 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:06.170895 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:06.170896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:06.170896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:06.171199 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:06.171379 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:06.171511 (+ 132us) server_negotiation.cc:299] Negotiation successful
0806 00:58:06.171666 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":150,"threads_started":1}
I20260806 00:58:06.172508 27081 catalog_manager.cc:1504] Loading token signing keys...
I20260806 00:58:06.183923 27081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe15af5ea2554c54ba97ba61059aa957: Generated new TSK 0
I20260806 00:58:06.184165 27081 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.242.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
I20260806 00:58:06.219779 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:0
--local_ip_for_outbound_sockets=127.23.242.1
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:39243
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:06.316004 27096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:06.316260 27096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:06.316291 27096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:06.317978 27096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:06.318051 27096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:06.318090 27096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 00:58:06.320695 27096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:39243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:06.321025 27096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:06.321336 27096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:06.327565 27104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:06.327622 27102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:06.327852 27101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:06.328197 27096 server_base.cc:1034] running on GCE node
I20260806 00:58:06.328431 27096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:06.329037 27096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:06.330292 27096 hybrid_clock.cc:648] HybridClock initialized: now 1785977886330238 us; error 90 us; skew 500 ppm
Aug 06 00:58:06 dist-test-slave-dhph krb5kdc[26986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977886, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:06.333964 27096 init.cc:377] Logged in from keytab as kudu/127.23.242.1@KRBTEST.COM (short username kudu)
I20260806 00:58:06.334993 27096 webserver.cc:460] Webserver started at http://127.23.242.1:40691/ using document root <none> and password file <none>
I20260806 00:58:06.335426 27096 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:06.335528 27096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:06.335753 27096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:06.337001 27096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/data/instance:
uuid: "02374140e9f141e08fadbeb0aad0e0ce"
format_stamp: "Formatted at 2026-08-06 00:58:06 on dist-test-slave-dhph"
server_key: "a04d94caeae42dfb18d6267e1fdb796e"
server_key_iv: "70fae7de6066cd372d1e499611a270f4"
server_key_version: "encryptionkey@0"
I20260806 00:58:06.337543 27096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance:
uuid: "02374140e9f141e08fadbeb0aad0e0ce"
format_stamp: "Formatted at 2026-08-06 00:58:06 on dist-test-slave-dhph"
server_key: "a04d94caeae42dfb18d6267e1fdb796e"
server_key_iv: "70fae7de6066cd372d1e499611a270f4"
server_key_version: "encryptionkey@0"
I20260806 00:58:06.339650 27096 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260806 00:58:06.341135 27111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:06.341542 27096 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260806 00:58:06.341755 27096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "02374140e9f141e08fadbeb0aad0e0ce"
format_stamp: "Formatted at 2026-08-06 00:58:06 on dist-test-slave-dhph"
server_key: "a04d94caeae42dfb18d6267e1fdb796e"
server_key_iv: "70fae7de6066cd372d1e499611a270f4"
server_key_version: "encryptionkey@0"
I20260806 00:58:06.341893 27096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:06.367165 27096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:06.370231 27096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:06.370440 27096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:06.370805 27096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:06.371181 27096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:06.371244 27096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:06.371294 27096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:06.371321 27096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:06.378527 27096 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:33881
I20260806 00:58:06.378772 27227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:33881 every 8 connection(s)
I20260806 00:58:06.379330 27096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 00:58:06.387420 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 27096
I20260806 00:58:06.387533 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance
Aug 06 00:58:06 dist-test-slave-dhph krb5kdc[26986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977886, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:06.387730 24520 external_mini_cluster.cc:1428] Setting key 8a67bee0c0ce07d132fc0c5435f15344
I20260806 00:58:06.391993 27072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:06.381252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:45769 (local address 127.23.242.62:39243)
0806 00:58:06.381445 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:06.381449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:06.381473 (+ 24us) server_negotiation.cc:407] Connection header received
0806 00:58:06.382064 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:06.382065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:06.382080 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:06.382159 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:06.383480 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:06.384086 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:06.385106 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:06.385324 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:06.385437 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:06.388236 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:06.388238 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:06.388238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:06.388246 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:06.390430 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:06.390894 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:06.390895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:06.390896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:06.390958 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:06.391236 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:06.391237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:06.391237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:06.391360 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:06.391436 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:06.391835 (+ 399us) server_negotiation.cc:299] Negotiation successful
0806 00:58:06.391924 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260806 00:58:06.392220 27231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:06.381052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:39243 (local address 127.23.242.1:45769)
0806 00:58:06.381358 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:06.381407 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:06.381913 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:06.382337 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:06.382342 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:06.383132 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:06.383354 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:06.383359 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:06.384230 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:06.384232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:06.385008 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:06.385010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:06.385498 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:06.385500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:06.385691 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:06.386368 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:06.386379 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:06.388117 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:06.390507 (+ 2390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:06.390516 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:06.390519 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:06.390811 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:06.391033 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:06.391034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:06.391034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:06.391159 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:06.391428 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:06.391435 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:06.391614 (+ 179us) client_negotiation.cc:769] Sending connection context
0806 00:58:06.391726 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 00:58:06.391912 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":132,"threads_started":1}
I20260806 00:58:06.392817 27228 heartbeater.cc:347] Connected to a master server at 127.23.242.62:39243
I20260806 00:58:06.392917 27228 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:06.393133 27228 heartbeater.cc:511] Master 127.23.242.62:39243 requested a full tablet report, sending...
I20260806 00:58:06.393702 27032 ts_manager.cc:194] Registered new tserver with Master: 02374140e9f141e08fadbeb0aad0e0ce (127.23.242.1:33881)
I20260806 00:58:06.394322 27032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.1@KRBTEST.COM'} at 127.23.242.1:45769
WARNING: no policy specified for kudu/127.23.242.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
I20260806 00:58:06.422430 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:0
--local_ip_for_outbound_sockets=127.23.242.2
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:39243
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:06.518486 27234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:06.518832 27234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:06.518870 27234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:06.520258 27234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:06.520331 27234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:06.520367 27234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 00:58:06.522128 27234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:39243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:06.522368 27234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:06.522658 27234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:06.528901 27239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:06.528808 27242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:06.528808 27240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:06.529431 27234 server_base.cc:1034] running on GCE node
I20260806 00:58:06.529645 27234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:06.529874 27234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:06.531052 27234 hybrid_clock.cc:648] HybridClock initialized: now 1785977886531031 us; error 34 us; skew 500 ppm
Aug 06 00:58:06 dist-test-slave-dhph krb5kdc[26986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977886, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:06.534204 27234 init.cc:377] Logged in from keytab as kudu/127.23.242.2@KRBTEST.COM (short username kudu)
I20260806 00:58:06.534837 27234 webserver.cc:460] Webserver started at http://127.23.242.2:38131/ using document root <none> and password file <none>
I20260806 00:58:06.535130 27234 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:06.535210 27234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:06.535382 27234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:06.536602 27234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/data/instance:
uuid: "2d33a6de52af4258a93677ccf7602006"
format_stamp: "Formatted at 2026-08-06 00:58:06 on dist-test-slave-dhph"
server_key: "df394e6feaf87f3b036e57bb33968675"
server_key_iv: "8783cf74384d238fe03a312853b5b33e"
server_key_version: "encryptionkey@0"
I20260806 00:58:06.537031 27234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance:
uuid: "2d33a6de52af4258a93677ccf7602006"
format_stamp: "Formatted at 2026-08-06 00:58:06 on dist-test-slave-dhph"
server_key: "df394e6feaf87f3b036e57bb33968675"
server_key_iv: "8783cf74384d238fe03a312853b5b33e"
server_key_version: "encryptionkey@0"
I20260806 00:58:06.538668 27234 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 00:58:06.539824 27249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:06.540123 27234 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:06.540241 27234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "2d33a6de52af4258a93677ccf7602006"
format_stamp: "Formatted at 2026-08-06 00:58:06 on dist-test-slave-dhph"
server_key: "df394e6feaf87f3b036e57bb33968675"
server_key_iv: "8783cf74384d238fe03a312853b5b33e"
server_key_version: "encryptionkey@0"
I20260806 00:58:06.540342 27234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:06.560977 27234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:06.563824 27234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:06.564026 27234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:06.564324 27234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:06.564689 27234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:06.564759 27234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:06.564807 27234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:06.564841 27234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:06.571979 27234 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:43015
I20260806 00:58:06.572027 27366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:43015 every 8 connection(s)
I20260806 00:58:06.572386 27234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
Aug 06 00:58:06 dist-test-slave-dhph krb5kdc[26986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977886, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:06.579901 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 27234
I20260806 00:58:06.580008 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance
I20260806 00:58:06.580214 24520 external_mini_cluster.cc:1428] Setting key f5136445c0d2551129447d9119bcac5f
I20260806 00:58:06.583714 27072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:06.573391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:53185 (local address 127.23.242.62:39243)
0806 00:58:06.573446 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:06.573448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:06.573728 (+ 280us) server_negotiation.cc:407] Connection header received
0806 00:58:06.574284 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:06.574286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:06.574297 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:06.574351 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:06.575448 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:06.576011 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:06.577199 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:06.577426 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:06.577499 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:06.580049 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:06.580051 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:06.580051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:06.580060 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:06.581927 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:06.582354 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:06.582355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:06.582356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:06.582419 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:06.582804 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:06.582805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:06.582805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:06.582977 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:06.583057 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:06.583499 (+ 442us) server_negotiation.cc:299] Negotiation successful
0806 00:58:06.583640 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 00:58:06.584021 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:06.573361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:39243 (local address 127.23.242.2:53185)
0806 00:58:06.573560 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:06.573603 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:06.574155 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:06.574416 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:06.574420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:06.575101 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:06.575320 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:06.575326 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:06.576169 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:06.576171 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:06.577071 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:06.577074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:06.577523 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:06.577524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:06.577640 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:06.578280 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:06.578289 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:06.579916 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:06.582024 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:06.582032 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:06.582034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:06.582272 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:06.582513 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:06.582514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:06.582514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:06.582726 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:06.583078 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:06.583086 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:06.583284 (+ 198us) client_negotiation.cc:769] Sending connection context
0806 00:58:06.583420 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 00:58:06.583648 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":97,"threads_started":1}
I20260806 00:58:06.584672 27367 heartbeater.cc:347] Connected to a master server at 127.23.242.62:39243
I20260806 00:58:06.584784 27367 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:06.585035 27367 heartbeater.cc:511] Master 127.23.242.62:39243 requested a full tablet report, sending...
I20260806 00:58:06.585522 27032 ts_manager.cc:194] Registered new tserver with Master: 2d33a6de52af4258a93677ccf7602006 (127.23.242.2:43015)
I20260806 00:58:06.586140 27032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.2@KRBTEST.COM'} at 127.23.242.2:53185
WARNING: no policy specified for kudu/127.23.242.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
I20260806 00:58:06.615443 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:0
--local_ip_for_outbound_sockets=127.23.242.3
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:39243
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:06.711930 27372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:06.712180 27372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:06.712209 27372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:06.713645 27372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:06.713713 27372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:06.713744 27372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 00:58:06.715431 27372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:39243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:06.715699 27372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:06.715972 27372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:06.721099 27378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:06.721109 27377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:06.721125 27380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:06.721468 27372 server_base.cc:1034] running on GCE node
I20260806 00:58:06.721755 27372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:06.722041 27372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:06.723248 27372 hybrid_clock.cc:648] HybridClock initialized: now 1785977886723221 us; error 46 us; skew 500 ppm
Aug 06 00:58:06 dist-test-slave-dhph krb5kdc[26986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977886, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:06.726210 27372 init.cc:377] Logged in from keytab as kudu/127.23.242.3@KRBTEST.COM (short username kudu)
I20260806 00:58:06.726820 27372 webserver.cc:460] Webserver started at http://127.23.242.3:42419/ using document root <none> and password file <none>
I20260806 00:58:06.727072 27372 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:06.727140 27372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:06.727293 27372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:06.728534 27372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/data/instance:
uuid: "ec603f31e1704136bd08216a11d239ed"
format_stamp: "Formatted at 2026-08-06 00:58:06 on dist-test-slave-dhph"
server_key: "7aa5ea7479ad24739a57080403c0ad15"
server_key_iv: "5d378eaf47a84a15e34e827b6fc1117e"
server_key_version: "encryptionkey@0"
I20260806 00:58:06.729079 27372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance:
uuid: "ec603f31e1704136bd08216a11d239ed"
format_stamp: "Formatted at 2026-08-06 00:58:06 on dist-test-slave-dhph"
server_key: "7aa5ea7479ad24739a57080403c0ad15"
server_key_iv: "5d378eaf47a84a15e34e827b6fc1117e"
server_key_version: "encryptionkey@0"
I20260806 00:58:06.730830 27372 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260806 00:58:06.731865 27387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:06.732234 27372 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:06.732393 27372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "ec603f31e1704136bd08216a11d239ed"
format_stamp: "Formatted at 2026-08-06 00:58:06 on dist-test-slave-dhph"
server_key: "7aa5ea7479ad24739a57080403c0ad15"
server_key_iv: "5d378eaf47a84a15e34e827b6fc1117e"
server_key_version: "encryptionkey@0"
I20260806 00:58:06.732487 27372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:06.746843 27372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:06.749621 27372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:06.749760 27372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:06.749971 27372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:06.750355 27372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:06.750432 27372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:06.750478 27372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:06.750532 27372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:06.757802 27372 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:36591
I20260806 00:58:06.757884 27503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:36591 every 8 connection(s)
I20260806 00:58:06.758271 27372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
I20260806 00:58:06.762176 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 27372
I20260806 00:58:06.762290 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance
I20260806 00:58:06.762673 24520 external_mini_cluster.cc:1428] Setting key 508fc05e53870e59b07d222e29ea873f
Aug 06 00:58:06 dist-test-slave-dhph krb5kdc[26986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977886, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:06.768887 27072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:06.759249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:41811 (local address 127.23.242.62:39243)
0806 00:58:06.759344 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:06.759347 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:06.759531 (+ 184us) server_negotiation.cc:407] Connection header received
0806 00:58:06.760161 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:06.760162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:06.760176 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:06.760232 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:06.761296 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:06.761858 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:06.763190 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:06.763482 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:06.763546 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:06.766059 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:06.766060 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:06.766061 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:06.766067 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:06.767513 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:06.767956 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:06.767957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:06.767957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:06.768017 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:06.768317 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:06.768317 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:06.768318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:06.768426 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:06.768492 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:06.768755 (+ 263us) server_negotiation.cc:299] Negotiation successful
0806 00:58:06.768821 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 00:58:06.769261 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:06.759127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:39243 (local address 127.23.242.3:41811)
0806 00:58:06.759378 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:06.759430 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:06.760025 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:06.760324 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:06.760328 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:06.760971 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:06.761189 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:06.761194 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:06.761979 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:06.761980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:06.763080 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:06.763084 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:06.763578 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:06.763579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:06.763761 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:06.764383 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:06.764392 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:06.765922 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:06.767607 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:06.767616 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:06.767618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:06.767864 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:06.768109 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:06.768110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:06.768110 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:06.768250 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:06.768496 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:06.768502 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:06.768621 (+ 119us) client_negotiation.cc:769] Sending connection context
0806 00:58:06.768725 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 00:58:06.768864 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":133,"threads_started":1}
I20260806 00:58:06.769837 27504 heartbeater.cc:347] Connected to a master server at 127.23.242.62:39243
I20260806 00:58:06.769937 27504 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:06.770174 27504 heartbeater.cc:511] Master 127.23.242.62:39243 requested a full tablet report, sending...
I20260806 00:58:06.770656 27032 ts_manager.cc:194] Registered new tserver with Master: ec603f31e1704136bd08216a11d239ed (127.23.242.3:36591)
I20260806 00:58:06.771091 27032 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.3@KRBTEST.COM'} at 127.23.242.3:41811
I20260806 00:58:06.774718 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 00:58:06.789237 27072 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:06.788171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36936 (local address 127.23.242.62:39243)
0806 00:58:06.788246 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:06.788248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:06.788327 (+ 79us) server_negotiation.cc:407] Connection header received
0806 00:58:06.788394 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:06.788395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:06.788405 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:06.788469 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:06.789178 (+ 709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36936: BlockingRecv error: recv got EOF from 127.0.0.1:36936 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 00:58:06.790153 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 27096
I20260806 00:58:06.795585 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 27234
I20260806 00:58:06.801419 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 27372
I20260806 00:58:06.807258 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 27000
2026-08-06T00:58:06Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2243 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 00:58:06 dist-test-slave-dhph krb5kdc[27516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 00:58:06 dist-test-slave-dhph krb5kdc[27516](info): set up 2 sockets
Aug 06 00:58:06 dist-test-slave-dhph krb5kdc[27516](info): commencing operation
krb5kdc: starting...
W20260806 00:58:08.014927 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.185s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 00:58:08 dist-test-slave-dhph krb5kdc[27516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T00:58:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T00:58:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 00:58:08.168083 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:43063
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:45289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:43063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:08.258358 27529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:08.258661 27529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:08.258741 27529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:08.259917 27529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 00:58:08.259987 27529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:08.260013 27529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:08.260046 27529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 00:58:08.260072 27529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 00:58:08.261626 27529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:45289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:43063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:43063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:08.261940 27529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:08.262181 27529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:08.267398 27535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:08.267434 27536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:08.267537 27529 server_base.cc:1034] running on GCE node
W20260806 00:58:08.267398 27538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:08.267938 27529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:08.268198 27529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:08.269371 27529 hybrid_clock.cc:648] HybridClock initialized: now 1785977888269359 us; error 33 us; skew 500 ppm
Aug 06 00:58:08 dist-test-slave-dhph krb5kdc[27516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977888, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:08.277486 27529 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 00:58:08.278069 27529 webserver.cc:460] Webserver started at http://127.23.242.62:44143/ using document root <none> and password file <none>
I20260806 00:58:08.278360 27529 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:08.278414 27529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:08.278568 27529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:08.279852 27529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "f503426018d146bd936aac1fdbfcc468"
format_stamp: "Formatted at 2026-08-06 00:58:08 on dist-test-slave-dhph"
server_key: "9245811ed6c1b648e5df913cf41e0474"
server_key_iv: "a2ef101b4573a286b09a3b4c64ae2f26"
server_key_version: "encryptionkey@0"
I20260806 00:58:08.280258 27529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "f503426018d146bd936aac1fdbfcc468"
format_stamp: "Formatted at 2026-08-06 00:58:08 on dist-test-slave-dhph"
server_key: "9245811ed6c1b648e5df913cf41e0474"
server_key_iv: "a2ef101b4573a286b09a3b4c64ae2f26"
server_key_version: "encryptionkey@0"
I20260806 00:58:08.281911 27529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 00:58:08.283313 27545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:08.283615 27529 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:08.283775 27529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "f503426018d146bd936aac1fdbfcc468"
format_stamp: "Formatted at 2026-08-06 00:58:08 on dist-test-slave-dhph"
server_key: "9245811ed6c1b648e5df913cf41e0474"
server_key_iv: "a2ef101b4573a286b09a3b4c64ae2f26"
server_key_version: "encryptionkey@0"
I20260806 00:58:08.283861 27529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:08.296423 27529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:08.299232 27529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:08.299427 27529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:08.304513 27529 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:43063
I20260806 00:58:08.304563 27599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:43063 every 8 connection(s)
I20260806 00:58:08.305161 27529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 00:58:08.305876 27600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:08.308449 27600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f503426018d146bd936aac1fdbfcc468: Bootstrap starting.
I20260806 00:58:08.309165 27600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f503426018d146bd936aac1fdbfcc468: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:08.309489 27600 log.cc:824] T 00000000000000000000000000000000 P f503426018d146bd936aac1fdbfcc468: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:08.310379 27600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f503426018d146bd936aac1fdbfcc468: No bootstrap required, opened a new log
I20260806 00:58:08.312325 27600 raft_consensus.cc:348] T 00000000000000000000000000000000 P f503426018d146bd936aac1fdbfcc468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f503426018d146bd936aac1fdbfcc468" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43063 } }
I20260806 00:58:08.312500 27600 raft_consensus.cc:374] T 00000000000000000000000000000000 P f503426018d146bd936aac1fdbfcc468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:08.312527 27600 raft_consensus.cc:729] T 00000000000000000000000000000000 P f503426018d146bd936aac1fdbfcc468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f503426018d146bd936aac1fdbfcc468, State: Initialized, Role: FOLLOWER
I20260806 00:58:08.312709 27600 consensus_queue.cc:260] T 00000000000000000000000000000000 P f503426018d146bd936aac1fdbfcc468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f503426018d146bd936aac1fdbfcc468" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43063 } }
I20260806 00:58:08.312799 27600 raft_consensus.cc:388] T 00000000000000000000000000000000 P f503426018d146bd936aac1fdbfcc468 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 00:58:08.312825 27600 raft_consensus.cc:482] T 00000000000000000000000000000000 P f503426018d146bd936aac1fdbfcc468 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 00:58:08.312918 27600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f503426018d146bd936aac1fdbfcc468 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:08.314339 27600 raft_consensus.cc:504] T 00000000000000000000000000000000 P f503426018d146bd936aac1fdbfcc468 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f503426018d146bd936aac1fdbfcc468" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43063 } }
I20260806 00:58:08.314451 27600 leader_election.cc:302] T 00000000000000000000000000000000 P f503426018d146bd936aac1fdbfcc468 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f503426018d146bd936aac1fdbfcc468; no voters:
I20260806 00:58:08.314692 27600 leader_election.cc:288] T 00000000000000000000000000000000 P f503426018d146bd936aac1fdbfcc468 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 00:58:08.314742 27603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f503426018d146bd936aac1fdbfcc468 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:08.314839 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 27529
I20260806 00:58:08.314935 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 00:58:08.314947 27603 raft_consensus.cc:686] T 00000000000000000000000000000000 P f503426018d146bd936aac1fdbfcc468 [term 1 LEADER]: Becoming Leader. State: Replica: f503426018d146bd936aac1fdbfcc468, State: Running, Role: LEADER
I20260806 00:58:08.315013 27600 sys_catalog.cc:564] T 00000000000000000000000000000000 P f503426018d146bd936aac1fdbfcc468 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 00:58:08.315042 27603 consensus_queue.cc:237] T 00000000000000000000000000000000 P f503426018d146bd936aac1fdbfcc468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f503426018d146bd936aac1fdbfcc468" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43063 } }
I20260806 00:58:08.315119 24520 external_mini_cluster.cc:1428] Setting key b86fab34fceb9c62cff5bb16de342e5e
I20260806 00:58:08.315529 27604 sys_catalog.cc:455] T 00000000000000000000000000000000 P f503426018d146bd936aac1fdbfcc468 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f503426018d146bd936aac1fdbfcc468" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f503426018d146bd936aac1fdbfcc468" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43063 } } }
I20260806 00:58:08.315619 27604 sys_catalog.cc:458] T 00000000000000000000000000000000 P f503426018d146bd936aac1fdbfcc468 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:08.315886 27605 sys_catalog.cc:455] T 00000000000000000000000000000000 P f503426018d146bd936aac1fdbfcc468 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f503426018d146bd936aac1fdbfcc468. Latest consensus state: current_term: 1 leader_uuid: "f503426018d146bd936aac1fdbfcc468" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f503426018d146bd936aac1fdbfcc468" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43063 } } }
I20260806 00:58:08.315951 27613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 00:58:08.316543 27613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 00:58:08.316699 27605 sys_catalog.cc:458] T 00000000000000000000000000000000 P f503426018d146bd936aac1fdbfcc468 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:08.318347 27613 catalog_manager.cc:1347] Generated new cluster ID: 3c2c23faf32b4a0a932fa334925cbbff
I20260806 00:58:08.318408 27613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 00:58:08 dist-test-slave-dhph krb5kdc[27516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:08.328390 27613 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 00:58:08.329125 27613 catalog_manager.cc:1504] Loading token signing keys...
I20260806 00:58:08.332065 27611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:08.315613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41932 (local address 127.23.242.62:43063)
0806 00:58:08.316119 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:08.316130 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:08.316172 (+ 42us) server_negotiation.cc:407] Connection header received
0806 00:58:08.316315 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:08.316322 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:08.316406 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:08.316573 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:08.317696 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:08.323408 (+ 5712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:08.324384 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:08.325035 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:08.325202 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:08.327815 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:08.327820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:08.327821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:08.327837 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:08.330068 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:08.330493 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:08.330497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:08.330497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:08.330653 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:08.330883 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:08.330884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:08.330884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:08.331185 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:08.331334 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:08.331443 (+ 109us) server_negotiation.cc:299] Negotiation successful
0806 00:58:08.331806 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":82,"threads_started":1}
I20260806 00:58:08.333053 24520 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 00:58:08.347311 27613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f503426018d146bd936aac1fdbfcc468: Generated new TSK 0
I20260806 00:58:08.347535 27613 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 00:58:08.350566 27611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:08.346803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41936 (local address 127.23.242.62:43063)
0806 00:58:08.346869 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:08.346871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:08.346973 (+ 102us) server_negotiation.cc:407] Connection header received
0806 00:58:08.347068 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:08.347069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:08.347080 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:08.347157 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:08.347711 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:08.348245 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:08.349006 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:08.349241 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:08.349336 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:08.350086 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:08.350087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:08.350088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 00:58:08.350295 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:08.350312 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 00:58:08.350319 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:08.350389 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:08.350445 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 00:58:08.350475 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 00:58:08.353755 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 27529
2026-08-06T00:58:08Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1543 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 00:58:08 dist-test-slave-dhph krb5kdc[27634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 00:58:08 dist-test-slave-dhph krb5kdc[27634](info): set up 2 sockets
Aug 06 00:58:08 dist-test-slave-dhph krb5kdc[27634](info): commencing operation
krb5kdc: starting...
W20260806 00:58:09.493659 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 00:58:09 dist-test-slave-dhph krb5kdc[27634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T00:58:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T00:58:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 00:58:09.644025 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:41393
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:33689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:41393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:09.737056 27647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:09.737391 27647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:09.737427 27647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:09.739194 27647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 00:58:09.739256 27647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:09.739272 27647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:09.739285 27647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 00:58:09.739298 27647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 00:58:09.740983 27647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:33689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:41393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:41393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:09.741215 27647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:09.741396 27647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:09.746775 27652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:09.746773 27653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:09.746764 27655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:09.746994 27647 server_base.cc:1034] running on GCE node
I20260806 00:58:09.747380 27647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:09.747694 27647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:09.749068 27647 hybrid_clock.cc:648] HybridClock initialized: now 1785977889749051 us; error 30 us; skew 500 ppm
Aug 06 00:58:09 dist-test-slave-dhph krb5kdc[27634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977889, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:09.751955 27647 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 00:58:09.752472 27647 webserver.cc:460] Webserver started at http://127.23.242.62:34199/ using document root <none> and password file <none>
I20260806 00:58:09.752730 27647 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:09.752808 27647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:09.752966 27647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:09.754057 27647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "9782057033204de6903dd8dbc3ae0f1a"
format_stamp: "Formatted at 2026-08-06 00:58:09 on dist-test-slave-dhph"
server_key: "b55142e5d2c78746c1cd7b8b31c33147"
server_key_iv: "0b2baaa8fd6d66934bba5f9702ddbe68"
server_key_version: "encryptionkey@0"
I20260806 00:58:09.754432 27647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "9782057033204de6903dd8dbc3ae0f1a"
format_stamp: "Formatted at 2026-08-06 00:58:09 on dist-test-slave-dhph"
server_key: "b55142e5d2c78746c1cd7b8b31c33147"
server_key_iv: "0b2baaa8fd6d66934bba5f9702ddbe68"
server_key_version: "encryptionkey@0"
I20260806 00:58:09.756028 27647 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:09.756911 27662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:09.757248 27647 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:09.757367 27647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "9782057033204de6903dd8dbc3ae0f1a"
format_stamp: "Formatted at 2026-08-06 00:58:09 on dist-test-slave-dhph"
server_key: "b55142e5d2c78746c1cd7b8b31c33147"
server_key_iv: "0b2baaa8fd6d66934bba5f9702ddbe68"
server_key_version: "encryptionkey@0"
I20260806 00:58:09.757483 27647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:09.776958 27647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:09.779737 27647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:09.779927 27647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:09.784778 27647 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:41393
I20260806 00:58:09.784821 27716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:41393 every 8 connection(s)
I20260806 00:58:09.785233 27647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 00:58:09.786355 27717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:09.788683 27717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9782057033204de6903dd8dbc3ae0f1a: Bootstrap starting.
I20260806 00:58:09.789220 27717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9782057033204de6903dd8dbc3ae0f1a: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:09.789456 27717 log.cc:824] T 00000000000000000000000000000000 P 9782057033204de6903dd8dbc3ae0f1a: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:09.790262 27717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9782057033204de6903dd8dbc3ae0f1a: No bootstrap required, opened a new log
I20260806 00:58:09.790480 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 27647
I20260806 00:58:09.790619 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 00:58:09.790823 24520 external_mini_cluster.cc:1428] Setting key 9f7b68cff8edad6cebe751a11be91b6d
I20260806 00:58:09.792735 27717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9782057033204de6903dd8dbc3ae0f1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9782057033204de6903dd8dbc3ae0f1a" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 41393 } }
I20260806 00:58:09.792922 27717 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9782057033204de6903dd8dbc3ae0f1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:09.792992 27717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9782057033204de6903dd8dbc3ae0f1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9782057033204de6903dd8dbc3ae0f1a, State: Initialized, Role: FOLLOWER
I20260806 00:58:09.793154 27717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9782057033204de6903dd8dbc3ae0f1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9782057033204de6903dd8dbc3ae0f1a" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 41393 } }
I20260806 00:58:09.793263 27717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9782057033204de6903dd8dbc3ae0f1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 00:58:09.793308 27717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9782057033204de6903dd8dbc3ae0f1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 00:58:09.793366 27717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9782057033204de6903dd8dbc3ae0f1a [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:09.794446 27717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9782057033204de6903dd8dbc3ae0f1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9782057033204de6903dd8dbc3ae0f1a" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 41393 } }
I20260806 00:58:09.794585 27717 leader_election.cc:302] T 00000000000000000000000000000000 P 9782057033204de6903dd8dbc3ae0f1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9782057033204de6903dd8dbc3ae0f1a; no voters:
I20260806 00:58:09.794826 27717 leader_election.cc:288] T 00000000000000000000000000000000 P 9782057033204de6903dd8dbc3ae0f1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 00:58:09.794858 27722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9782057033204de6903dd8dbc3ae0f1a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:09.794989 27722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9782057033204de6903dd8dbc3ae0f1a [term 1 LEADER]: Becoming Leader. State: Replica: 9782057033204de6903dd8dbc3ae0f1a, State: Running, Role: LEADER
I20260806 00:58:09.795140 27717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9782057033204de6903dd8dbc3ae0f1a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 00:58:09.795140 27722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9782057033204de6903dd8dbc3ae0f1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9782057033204de6903dd8dbc3ae0f1a" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 41393 } }
I20260806 00:58:09.795663 27723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9782057033204de6903dd8dbc3ae0f1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9782057033204de6903dd8dbc3ae0f1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9782057033204de6903dd8dbc3ae0f1a" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 41393 } } }
I20260806 00:58:09.795672 27724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9782057033204de6903dd8dbc3ae0f1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9782057033204de6903dd8dbc3ae0f1a. Latest consensus state: current_term: 1 leader_uuid: "9782057033204de6903dd8dbc3ae0f1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9782057033204de6903dd8dbc3ae0f1a" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 41393 } } }
I20260806 00:58:09.795859 27723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9782057033204de6903dd8dbc3ae0f1a [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:09.795871 27724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9782057033204de6903dd8dbc3ae0f1a [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:09.796175 27731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 00:58:09.796859 27731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 00:58:09.799086 27731 catalog_manager.cc:1347] Generated new cluster ID: 3be2d0ce738a417ab8098e3fe527f16a
I20260806 00:58:09.799171 27731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 00:58:09 dist-test-slave-dhph krb5kdc[27634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:09.806360 27721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:09.791594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34326 (local address 127.23.242.62:41393)
0806 00:58:09.791781 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:09.791790 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:09.791817 (+ 27us) server_negotiation.cc:407] Connection header received
0806 00:58:09.792759 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:09.792768 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:09.792823 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:09.792976 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:09.793835 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:09.794962 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:09.796262 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:09.796943 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:09.797568 (+ 625us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:09.801696 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:09.801707 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:09.801708 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:09.801720 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:09.804101 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:09.804575 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:09.804584 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:09.804584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:09.804708 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:09.804949 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:09.804950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:09.804950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:09.805227 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:09.805986 (+ 759us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:09.806052 (+ 66us) server_negotiation.cc:299] Negotiation successful
0806 00:58:09.806177 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
I20260806 00:58:09.812981 27731 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 00:58:09.813658 27731 catalog_manager.cc:1504] Loading token signing keys...
I20260806 00:58:09.821661 27731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9782057033204de6903dd8dbc3ae0f1a: Generated new TSK 0
I20260806 00:58:09.821907 27731 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.242.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
I20260806 00:58:09.865016 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:0
--local_ip_for_outbound_sockets=127.23.242.1
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:41393
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:33689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:09.970180 27743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:09.970413 27743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:09.970440 27743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:09.971709 27743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:09.971760 27743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:09.971794 27743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 00:58:09.973685 27743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:33689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:41393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:09.974030 27743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:09.974339 27743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:09.981292 27748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:09.981292 27751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:09.981369 27749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:09.981969 27743 server_base.cc:1034] running on GCE node
I20260806 00:58:09.982242 27743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:09.982546 27743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:09.984303 27743 hybrid_clock.cc:648] HybridClock initialized: now 1785977889984269 us; error 46 us; skew 500 ppm
Aug 06 00:58:09 dist-test-slave-dhph krb5kdc[27634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977889, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:09.988075 27743 init.cc:377] Logged in from keytab as kudu/127.23.242.1@KRBTEST.COM (short username kudu)
I20260806 00:58:09.988739 27743 webserver.cc:460] Webserver started at http://127.23.242.1:46115/ using document root <none> and password file <none>
I20260806 00:58:09.989064 27743 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:09.989156 27743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:09.989342 27743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:09.990705 27743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/data/instance:
uuid: "51f5bb90a71f4d7e960471fe4fd9903f"
format_stamp: "Formatted at 2026-08-06 00:58:09 on dist-test-slave-dhph"
server_key: "2df5416b9486cdcd5a01966ba6013cdf"
server_key_iv: "a88ab87a91591df9570909deb39c7f80"
server_key_version: "encryptionkey@0"
I20260806 00:58:09.991801 27743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance:
uuid: "51f5bb90a71f4d7e960471fe4fd9903f"
format_stamp: "Formatted at 2026-08-06 00:58:09 on dist-test-slave-dhph"
server_key: "2df5416b9486cdcd5a01966ba6013cdf"
server_key_iv: "a88ab87a91591df9570909deb39c7f80"
server_key_version: "encryptionkey@0"
I20260806 00:58:09.993933 27743 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260806 00:58:09.995162 27759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:09.995743 27743 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 00:58:09.995913 27743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "51f5bb90a71f4d7e960471fe4fd9903f"
format_stamp: "Formatted at 2026-08-06 00:58:09 on dist-test-slave-dhph"
server_key: "2df5416b9486cdcd5a01966ba6013cdf"
server_key_iv: "a88ab87a91591df9570909deb39c7f80"
server_key_version: "encryptionkey@0"
I20260806 00:58:09.996044 27743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:10.006275 27743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:10.011000 27743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:10.011243 27743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:10.011541 27743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:10.011927 27743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:10.011991 27743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:10.012042 27743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:10.012079 27743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:10.020994 27743 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:43347
I20260806 00:58:10.021101 27877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:43347 every 8 connection(s)
I20260806 00:58:10.021526 27743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 00:58:10.023240 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 27743
I20260806 00:58:10.023346 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance
I20260806 00:58:10.023780 24520 external_mini_cluster.cc:1428] Setting key 07df6b41beace7e7702bbc418c2b16f5
Aug 06 00:58:10 dist-test-slave-dhph krb5kdc[27634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977889, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.242.2@KRBTEST.COM; defaulting to no policy
I20260806 00:58:10.035631 27721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:10.022468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:47617 (local address 127.23.242.62:41393)
0806 00:58:10.022540 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:10.022542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:10.023055 (+ 513us) server_negotiation.cc:407] Connection header received
0806 00:58:10.023489 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:10.023490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:10.023588 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:10.023789 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:10.025346 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:10.026162 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:10.028095 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:10.028570 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:10.028661 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:10.031771 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:10.031772 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:10.031773 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:10.031783 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:10.033886 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:10.034409 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:10.034417 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:10.034417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:10.034480 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:10.034817 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:10.034818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:10.034819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:10.034961 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:10.035050 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:10.035385 (+ 335us) server_negotiation.cc:299] Negotiation successful
0806 00:58:10.035507 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 00:58:10.035844 27880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:10.022417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:41393 (local address 127.23.242.1:47617)
0806 00:58:10.022660 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:10.022710 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:10.023281 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:10.024344 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:10.024348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:10.025019 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:10.025233 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:10.025238 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:10.026404 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:10.026405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:10.027903 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:10.027907 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:10.028696 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:10.028697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:10.028874 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:10.029749 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:10.029762 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:10.031640 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:10.034007 (+ 2367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:10.034018 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:10.034022 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:10.034303 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:10.034566 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:10.034567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:10.034568 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:10.034722 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:10.035040 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:10.035048 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:10.035222 (+ 174us) client_negotiation.cc:769] Sending connection context
0806 00:58:10.035343 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 00:58:10.035542 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":109,"threads_started":1}
I20260806 00:58:10.036568 27878 heartbeater.cc:347] Connected to a master server at 127.23.242.62:41393
I20260806 00:58:10.036686 27878 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:10.036926 27878 heartbeater.cc:511] Master 127.23.242.62:41393 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.2@KRBTEST.COM" created.
I20260806 00:58:10.037643 27681 ts_manager.cc:194] Registered new tserver with Master: 51f5bb90a71f4d7e960471fe4fd9903f (127.23.242.1:43347)
I20260806 00:58:10.038426 27681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.1@KRBTEST.COM'} at 127.23.242.1:47617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
I20260806 00:58:10.060708 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:0
--local_ip_for_outbound_sockets=127.23.242.2
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:41393
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:33689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:10.160101 27883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:10.160367 27883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:10.160403 27883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:10.162395 27883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:10.162482 27883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:10.162549 27883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 00:58:10.164705 27883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:33689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:41393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:10.165051 27883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:10.165369 27883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:10.171555 27889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:10.171824 27888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:10.171824 27891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:10.172585 27883 server_base.cc:1034] running on GCE node
I20260806 00:58:10.172848 27883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:10.173097 27883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:10.174522 27883 hybrid_clock.cc:648] HybridClock initialized: now 1785977890174521 us; error 74 us; skew 500 ppm
Aug 06 00:58:10 dist-test-slave-dhph krb5kdc[27634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977890, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:10.178223 27883 init.cc:377] Logged in from keytab as kudu/127.23.242.2@KRBTEST.COM (short username kudu)
I20260806 00:58:10.178953 27883 webserver.cc:460] Webserver started at http://127.23.242.2:41097/ using document root <none> and password file <none>
I20260806 00:58:10.179250 27883 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:10.179332 27883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:10.179504 27883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:10.180665 27883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/data/instance:
uuid: "bb0a509b022f474db9005910b278d32d"
format_stamp: "Formatted at 2026-08-06 00:58:10 on dist-test-slave-dhph"
server_key: "accd290d830688b69ad8c6867cd95a2c"
server_key_iv: "29c0a3d5a0e1d30bd4816d4123be1874"
server_key_version: "encryptionkey@0"
I20260806 00:58:10.181077 27883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance:
uuid: "bb0a509b022f474db9005910b278d32d"
format_stamp: "Formatted at 2026-08-06 00:58:10 on dist-test-slave-dhph"
server_key: "accd290d830688b69ad8c6867cd95a2c"
server_key_iv: "29c0a3d5a0e1d30bd4816d4123be1874"
server_key_version: "encryptionkey@0"
I20260806 00:58:10.182732 27883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260806 00:58:10.183911 27898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:10.184386 27883 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:10.184526 27883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "bb0a509b022f474db9005910b278d32d"
format_stamp: "Formatted at 2026-08-06 00:58:10 on dist-test-slave-dhph"
server_key: "accd290d830688b69ad8c6867cd95a2c"
server_key_iv: "29c0a3d5a0e1d30bd4816d4123be1874"
server_key_version: "encryptionkey@0"
I20260806 00:58:10.184621 27883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:10.224088 27883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:10.227128 27883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:10.227334 27883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:10.227634 27883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:10.228036 27883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:10.228094 27883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:10.228142 27883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:10.228173 27883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:10.235159 27883 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:46661
I20260806 00:58:10.235209 28015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:46661 every 8 connection(s)
I20260806 00:58:10.235617 27883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 00:58:10.238265 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 27883
I20260806 00:58:10.238417 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance
I20260806 00:58:10.238648 24520 external_mini_cluster.cc:1428] Setting key 86e70327a92ca29cb0f2ecac56f37006
Aug 06 00:58:10 dist-test-slave-dhph krb5kdc[27634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977890, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:10.247246 27721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:10.236575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:52305 (local address 127.23.242.62:41393)
0806 00:58:10.236659 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:10.236662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:10.236975 (+ 313us) server_negotiation.cc:407] Connection header received
0806 00:58:10.237541 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:10.237542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:10.237555 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:10.237605 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:10.238673 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:10.239302 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:10.240913 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:10.241191 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:10.241269 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:10.244048 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:10.244050 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:10.244050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:10.244060 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:10.245661 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:10.246122 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:10.246124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:10.246124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:10.246189 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:10.246484 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:10.246484 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:10.246485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:10.246671 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:10.246756 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:10.247042 (+ 286us) server_negotiation.cc:299] Negotiation successful
0806 00:58:10.247149 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 00:58:10.247466 28020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:10.236537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:41393 (local address 127.23.242.2:52305)
0806 00:58:10.236826 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:10.236880 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:10.237426 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:10.237684 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:10.237688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:10.238292 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:10.238559 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:10.238564 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:10.239435 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:10.239436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:10.240797 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:10.240800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:10.241313 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:10.241314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:10.241477 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:10.242100 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:10.242112 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:10.243927 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:10.245778 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:10.245788 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:10.245791 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:10.246034 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:10.246279 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:10.246280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:10.246281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:10.246392 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:10.246746 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:10.246754 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:10.246903 (+ 149us) client_negotiation.cc:769] Sending connection context
0806 00:58:10.247021 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 00:58:10.247195 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for kudu/127.23.242.3@KRBTEST.COM; defaulting to no policy
I20260806 00:58:10.248170 28016 heartbeater.cc:347] Connected to a master server at 127.23.242.62:41393
I20260806 00:58:10.248315 28016 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:10.248580 28016 heartbeater.cc:511] Master 127.23.242.62:41393 requested a full tablet report, sending...
I20260806 00:58:10.249073 27681 ts_manager.cc:194] Registered new tserver with Master: bb0a509b022f474db9005910b278d32d (127.23.242.2:46661)
I20260806 00:58:10.249579 27681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.2@KRBTEST.COM'} at 127.23.242.2:52305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
I20260806 00:58:10.275905 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:0
--local_ip_for_outbound_sockets=127.23.242.3
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:41393
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:33689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:10.372886 28023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:10.373116 28023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:10.373148 28023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:10.374321 28023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:10.374363 28023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:10.374392 28023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 00:58:10.375859 28023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:33689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:41393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:10.376065 28023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:10.376266 28023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:10.382066 28028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:10.382066 28031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:10.382108 28029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:10.382791 28023 server_base.cc:1034] running on GCE node
I20260806 00:58:10.383033 28023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:10.383430 28023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:10.384703 28023 hybrid_clock.cc:648] HybridClock initialized: now 1785977890384658 us; error 61 us; skew 500 ppm
Aug 06 00:58:10 dist-test-slave-dhph krb5kdc[27634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977890, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:10.388023 28023 init.cc:377] Logged in from keytab as kudu/127.23.242.3@KRBTEST.COM (short username kudu)
I20260806 00:58:10.388523 28023 webserver.cc:460] Webserver started at http://127.23.242.3:44157/ using document root <none> and password file <none>
I20260806 00:58:10.388798 28023 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:10.388857 28023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:10.389011 28023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:10.390376 28023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/data/instance:
uuid: "ed92f668bc324605ab4ff2dfad05555a"
format_stamp: "Formatted at 2026-08-06 00:58:10 on dist-test-slave-dhph"
server_key: "8e2d5333fb1c32b58757b50b8431d83b"
server_key_iv: "540c4bbf549180e2bf71ef863c78eec8"
server_key_version: "encryptionkey@0"
I20260806 00:58:10.390882 28023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance:
uuid: "ed92f668bc324605ab4ff2dfad05555a"
format_stamp: "Formatted at 2026-08-06 00:58:10 on dist-test-slave-dhph"
server_key: "8e2d5333fb1c32b58757b50b8431d83b"
server_key_iv: "540c4bbf549180e2bf71ef863c78eec8"
server_key_version: "encryptionkey@0"
I20260806 00:58:10.392587 28023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 00:58:10.393666 28038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:10.393982 28023 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:10.394073 28023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "ed92f668bc324605ab4ff2dfad05555a"
format_stamp: "Formatted at 2026-08-06 00:58:10 on dist-test-slave-dhph"
server_key: "8e2d5333fb1c32b58757b50b8431d83b"
server_key_iv: "540c4bbf549180e2bf71ef863c78eec8"
server_key_version: "encryptionkey@0"
I20260806 00:58:10.394214 28023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:10.418322 28023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:10.420965 28023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:10.421109 28023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:10.421451 28023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:10.421801 28023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:10.421866 28023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:10.421914 28023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:10.421943 28023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:10.428851 28023 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:42785
I20260806 00:58:10.428957 28154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:42785 every 8 connection(s)
I20260806 00:58:10.429294 28023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
I20260806 00:58:10.433092 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 28023
I20260806 00:58:10.433228 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance
I20260806 00:58:10.433403 24520 external_mini_cluster.cc:1428] Setting key a4077919d136189fad7d9f21ae1bf211
Aug 06 00:58:10 dist-test-slave-dhph krb5kdc[27634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977890, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:10.441169 27721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:10.430318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:36243 (local address 127.23.242.62:41393)
0806 00:58:10.430376 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:10.430378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:10.430867 (+ 489us) server_negotiation.cc:407] Connection header received
0806 00:58:10.431556 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:10.431557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:10.431569 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:10.431624 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:10.432919 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:10.433712 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:10.434978 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:10.435196 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:10.435284 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:10.437953 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:10.437955 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:10.437955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:10.437964 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:10.439668 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:10.440164 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:10.440166 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:10.440166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:10.440231 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:10.440498 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:10.440499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:10.440499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:10.440629 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:10.440714 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:10.440988 (+ 274us) server_negotiation.cc:299] Negotiation successful
0806 00:58:10.441072 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 00:58:10.441406 28158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:10.430158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:41393 (local address 127.23.242.3:36243)
0806 00:58:10.430443 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:10.430680 (+ 237us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:10.431284 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:10.431749 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:10.431753 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:10.432537 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:10.432786 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:10.432792 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:10.433826 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:10.433827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:10.434886 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:10.434890 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:10.435298 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:10.435300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:10.435422 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:10.436165 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:10.436174 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:10.437836 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:10.439817 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:10.439826 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:10.439828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:10.440069 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:10.440319 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:10.440320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:10.440321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:10.440429 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:10.440704 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:10.440711 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:10.440852 (+ 141us) client_negotiation.cc:769] Sending connection context
0806 00:58:10.440964 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 00:58:10.441132 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":110,"threads_started":1}
I20260806 00:58:10.442013 28155 heartbeater.cc:347] Connected to a master server at 127.23.242.62:41393
I20260806 00:58:10.442128 28155 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:10.442356 28155 heartbeater.cc:511] Master 127.23.242.62:41393 requested a full tablet report, sending...
I20260806 00:58:10.442938 27681 ts_manager.cc:194] Registered new tserver with Master: ed92f668bc324605ab4ff2dfad05555a (127.23.242.3:42785)
I20260806 00:58:10.443347 27681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.3@KRBTEST.COM'} at 127.23.242.3:36243
I20260806 00:58:10.446177 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 00:58:10 dist-test-slave-dhph krb5kdc[27634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977890, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 00:58:10 dist-test-slave-dhph krb5kdc[27634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977890, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:10.476225 27721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:10.467046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34340 (local address 127.23.242.62:41393)
0806 00:58:10.467164 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:10.467166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:10.467286 (+ 120us) server_negotiation.cc:407] Connection header received
0806 00:58:10.467323 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:10.467324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:10.467337 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:10.467403 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:10.468137 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:10.468709 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:10.469618 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:10.469838 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:10.469910 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:10.472828 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:10.472829 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:10.472830 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:10.472837 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:10.474489 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:10.474917 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:10.474918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:10.474918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:10.474981 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:10.475406 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:10.475407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:10.475407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:10.475593 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:10.475736 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:10.475963 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 00:58:10.476049 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260806 00:58:10.476595 27681 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34340
I20260806 00:58:10.477316 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 27743
I20260806 00:58:10.483162 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 27883
I20260806 00:58:10.488677 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 28023
I20260806 00:58:10.494710 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 27647
2026-08-06T00:58:10Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2143 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 00:58:10 dist-test-slave-dhph krb5kdc[28168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 00:58:10 dist-test-slave-dhph krb5kdc[28168](info): set up 2 sockets
Aug 06 00:58:10 dist-test-slave-dhph krb5kdc[28168](info): commencing operation
krb5kdc: starting...
W20260806 00:58:11.671200 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 00:58:11 dist-test-slave-dhph krb5kdc[28168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T00:58:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T00:58:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 00:58:11.811292 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:45273
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:41071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:45273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:11.901731 28181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:11.902019 28181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:11.902058 28181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:11.903282 28181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 00:58:11.903357 28181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:11.903381 28181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:11.903401 28181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 00:58:11.903456 28181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 00:58:11.905313 28181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:45273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:45273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:11.905615 28181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:11.905864 28181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:11.911617 28190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:11.911607 28188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:11.911896 28181 server_base.cc:1034] running on GCE node
W20260806 00:58:11.911609 28187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:11.912286 28181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:11.912632 28181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:11.913874 28181 hybrid_clock.cc:648] HybridClock initialized: now 1785977891913834 us; error 52 us; skew 500 ppm
Aug 06 00:58:11 dist-test-slave-dhph krb5kdc[28168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977891, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:11.917084 28181 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 00:58:11.917760 28181 webserver.cc:460] Webserver started at http://127.23.242.62:36573/ using document root <none> and password file <none>
I20260806 00:58:11.918008 28181 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:11.918079 28181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:11.918237 28181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:11.919435 28181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "f4ee116c299e4014a83165c0d709a3a4"
format_stamp: "Formatted at 2026-08-06 00:58:11 on dist-test-slave-dhph"
server_key: "1780f2c0896df626986933e6adb4a3c4"
server_key_iv: "3fd10bfc25c1594f7893e2493d5e01a9"
server_key_version: "encryptionkey@0"
I20260806 00:58:11.919826 28181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "f4ee116c299e4014a83165c0d709a3a4"
format_stamp: "Formatted at 2026-08-06 00:58:11 on dist-test-slave-dhph"
server_key: "1780f2c0896df626986933e6adb4a3c4"
server_key_iv: "3fd10bfc25c1594f7893e2493d5e01a9"
server_key_version: "encryptionkey@0"
I20260806 00:58:11.921367 28181 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 00:58:11.922686 28198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:11.922948 28181 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:11.923069 28181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "f4ee116c299e4014a83165c0d709a3a4"
format_stamp: "Formatted at 2026-08-06 00:58:11 on dist-test-slave-dhph"
server_key: "1780f2c0896df626986933e6adb4a3c4"
server_key_iv: "3fd10bfc25c1594f7893e2493d5e01a9"
server_key_version: "encryptionkey@0"
I20260806 00:58:11.923264 28181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:11.945204 28181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:11.948195 28181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:11.948412 28181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:11.953727 28181 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:45273
I20260806 00:58:11.953780 28254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:45273 every 8 connection(s)
I20260806 00:58:11.954380 28181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 00:58:11.955221 28255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:11.957765 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 28181
I20260806 00:58:11.957880 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 00:58:11.958091 24520 external_mini_cluster.cc:1428] Setting key 3daad8eaa347dc0cb24319cc879e89ee
I20260806 00:58:11.958874 28255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4ee116c299e4014a83165c0d709a3a4: Bootstrap starting.
I20260806 00:58:11.959707 28255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4ee116c299e4014a83165c0d709a3a4: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:11.960140 28255 log.cc:824] T 00000000000000000000000000000000 P f4ee116c299e4014a83165c0d709a3a4: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:11.961154 28255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4ee116c299e4014a83165c0d709a3a4: No bootstrap required, opened a new log
I20260806 00:58:11.964406 28255 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4ee116c299e4014a83165c0d709a3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ee116c299e4014a83165c0d709a3a4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 45273 } }
I20260806 00:58:11.964577 28255 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4ee116c299e4014a83165c0d709a3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:11.964603 28255 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4ee116c299e4014a83165c0d709a3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4ee116c299e4014a83165c0d709a3a4, State: Initialized, Role: FOLLOWER
I20260806 00:58:11.964762 28255 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4ee116c299e4014a83165c0d709a3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ee116c299e4014a83165c0d709a3a4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 45273 } }
I20260806 00:58:11.964848 28255 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4ee116c299e4014a83165c0d709a3a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 00:58:11.964905 28255 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4ee116c299e4014a83165c0d709a3a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 00:58:11.964955 28255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4ee116c299e4014a83165c0d709a3a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:11.965871 28255 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4ee116c299e4014a83165c0d709a3a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ee116c299e4014a83165c0d709a3a4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 45273 } }
I20260806 00:58:11.966002 28255 leader_election.cc:302] T 00000000000000000000000000000000 P f4ee116c299e4014a83165c0d709a3a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4ee116c299e4014a83165c0d709a3a4; no voters:
I20260806 00:58:11.966216 28255 leader_election.cc:288] T 00000000000000000000000000000000 P f4ee116c299e4014a83165c0d709a3a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 00:58:11.966320 28260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4ee116c299e4014a83165c0d709a3a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:11.966558 28255 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4ee116c299e4014a83165c0d709a3a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 00:58:11.966564 28260 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4ee116c299e4014a83165c0d709a3a4 [term 1 LEADER]: Becoming Leader. State: Replica: f4ee116c299e4014a83165c0d709a3a4, State: Running, Role: LEADER
I20260806 00:58:11.966907 28260 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4ee116c299e4014a83165c0d709a3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ee116c299e4014a83165c0d709a3a4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 45273 } }
I20260806 00:58:11.967321 28261 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4ee116c299e4014a83165c0d709a3a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4ee116c299e4014a83165c0d709a3a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ee116c299e4014a83165c0d709a3a4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 45273 } } }
I20260806 00:58:11.967406 28261 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4ee116c299e4014a83165c0d709a3a4 [sys.catalog]: This master's current role is: LEADER
Aug 06 00:58:11 dist-test-slave-dhph krb5kdc[28168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:11.967726 28268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 00:58:11.967799 28262 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4ee116c299e4014a83165c0d709a3a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4ee116c299e4014a83165c0d709a3a4. Latest consensus state: current_term: 1 leader_uuid: "f4ee116c299e4014a83165c0d709a3a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4ee116c299e4014a83165c0d709a3a4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 45273 } } }
I20260806 00:58:11.967877 28262 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4ee116c299e4014a83165c0d709a3a4 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:11.968271 28268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 00:58:11.971025 28268 catalog_manager.cc:1347] Generated new cluster ID: 4db7d2d3986642bd9848461e2d458f18
I20260806 00:58:11.971107 28268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 00:58:11.973694 28258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:11.958802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54610 (local address 127.23.242.62:45273)
0806 00:58:11.959000 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:11.959018 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:11.959047 (+ 29us) server_negotiation.cc:407] Connection header received
0806 00:58:11.959539 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:11.959546 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:11.959596 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:11.959723 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:11.960678 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:11.961802 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:11.963726 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:11.964912 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:11.965057 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:11.968216 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:11.968221 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:11.968221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:11.968236 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:11.971487 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:11.971975 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:11.971979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:11.971979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:11.972166 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:11.972444 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:11.972445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:11.972445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:11.972980 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:11.973138 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:11.973267 (+ 129us) server_negotiation.cc:299] Negotiation successful
0806 00:58:11.973462 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":103,"threads_started":1}
I20260806 00:58:11.987931 28268 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 00:58:11.988678 28268 catalog_manager.cc:1504] Loading token signing keys...
I20260806 00:58:11.994812 28268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4ee116c299e4014a83165c0d709a3a4: Generated new TSK 0
I20260806 00:58:11.995158 28268 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.242.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
I20260806 00:58:12.036037 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:0
--local_ip_for_outbound_sockets=127.23.242.1
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:45273
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:41071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:12.141918 28281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:12.142184 28281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:12.142213 28281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:12.143852 28281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:12.143918 28281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:12.143951 28281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 00:58:12.145722 28281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:45273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:12.146008 28281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:12.146309 28281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:12.153097 28286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:12.153090 28287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:12.153090 28289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:12.153362 28281 server_base.cc:1034] running on GCE node
I20260806 00:58:12.153880 28281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:12.154158 28281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:12.155540 28281 hybrid_clock.cc:648] HybridClock initialized: now 1785977892155510 us; error 31 us; skew 500 ppm
Aug 06 00:58:12 dist-test-slave-dhph krb5kdc[28168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977892, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:12.159024 28281 init.cc:377] Logged in from keytab as kudu/127.23.242.1@KRBTEST.COM (short username kudu)
I20260806 00:58:12.159693 28281 webserver.cc:460] Webserver started at http://127.23.242.1:34339/ using document root <none> and password file <none>
I20260806 00:58:12.159962 28281 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:12.160041 28281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:12.160336 28281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:12.161501 28281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-0/data/instance:
uuid: "bdea875343ca4532b075183ea24e0df6"
format_stamp: "Formatted at 2026-08-06 00:58:12 on dist-test-slave-dhph"
server_key: "167b317b9c236201006d82a45943a67e"
server_key_iv: "afebf1c843fb5658f88b77a19cef1fd1"
server_key_version: "encryptionkey@0"
I20260806 00:58:12.161891 28281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance:
uuid: "bdea875343ca4532b075183ea24e0df6"
format_stamp: "Formatted at 2026-08-06 00:58:12 on dist-test-slave-dhph"
server_key: "167b317b9c236201006d82a45943a67e"
server_key_iv: "afebf1c843fb5658f88b77a19cef1fd1"
server_key_version: "encryptionkey@0"
I20260806 00:58:12.163663 28281 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 00:58:12.164660 28296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:12.164934 28281 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:12.165067 28281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "bdea875343ca4532b075183ea24e0df6"
format_stamp: "Formatted at 2026-08-06 00:58:12 on dist-test-slave-dhph"
server_key: "167b317b9c236201006d82a45943a67e"
server_key_iv: "afebf1c843fb5658f88b77a19cef1fd1"
server_key_version: "encryptionkey@0"
I20260806 00:58:12.165163 28281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:12.174295 28281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:12.176838 28281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:12.177021 28281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:12.177277 28281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:12.177614 28281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:12.177670 28281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:12.177745 28281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:12.177775 28281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:12.185822 28281 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:34039
I20260806 00:58:12.185854 28415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:34039 every 8 connection(s)
I20260806 00:58:12.186477 28281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 00:58:12.194170 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 28281
I20260806 00:58:12.194290 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance
I20260806 00:58:12.194514 24520 external_mini_cluster.cc:1428] Setting key 3c511b51b609482b2a47a88e73698c54
Aug 06 00:58:12 dist-test-slave-dhph krb5kdc[28168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977892, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:12.199836 28258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:12.187558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:33529 (local address 127.23.242.62:45273)
0806 00:58:12.187695 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:12.187697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:12.187902 (+ 205us) server_negotiation.cc:407] Connection header received
0806 00:58:12.188896 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:12.188897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:12.188910 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:12.188975 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:12.189643 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:12.190176 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:12.191615 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:12.191835 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:12.191914 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:12.195553 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:12.195556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:12.195556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:12.195569 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:12.197573 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:12.198424 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:12.198425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:12.198425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:12.198503 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:12.198853 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:12.198854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:12.198854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:12.199019 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:12.199098 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:12.199533 (+ 435us) server_negotiation.cc:299] Negotiation successful
0806 00:58:12.199687 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260806 00:58:12.199828 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:12.187472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45273 (local address 127.23.242.1:33529)
0806 00:58:12.187767 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:12.187806 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:12.188778 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:12.189049 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:12.189052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:12.189327 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:12.189525 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:12.189529 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:12.190289 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:12.190290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:12.191471 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:12.191474 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:12.191924 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:12.191925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:12.192064 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:12.192848 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:12.192863 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:12.195387 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:12.197726 (+ 2339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:12.197742 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:12.197745 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:12.198310 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:12.198639 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:12.198640 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:12.198640 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:12.198778 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:12.199133 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:12.199141 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:12.199322 (+ 181us) client_negotiation.cc:769] Sending connection context
0806 00:58:12.199456 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 00:58:12.199642 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":98,"threads_started":1}
I20260806 00:58:12.200562 28416 heartbeater.cc:347] Connected to a master server at 127.23.242.62:45273
I20260806 00:58:12.200693 28416 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:12.200894 28416 heartbeater.cc:511] Master 127.23.242.62:45273 requested a full tablet report, sending...
I20260806 00:58:12.201505 28216 ts_manager.cc:194] Registered new tserver with Master: bdea875343ca4532b075183ea24e0df6 (127.23.242.1:34039)
I20260806 00:58:12.202505 28216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.1@KRBTEST.COM'} at 127.23.242.1:33529
WARNING: no policy specified for kudu/127.23.242.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
I20260806 00:58:12.227640 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:0
--local_ip_for_outbound_sockets=127.23.242.2
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:45273
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:41071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:12.320940 28421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:12.321164 28421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:12.321192 28421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:12.322373 28421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:12.322417 28421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:12.322444 28421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 00:58:12.324144 28421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:45273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:12.324347 28421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:12.324527 28421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:12.330251 28426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:12.330260 28429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:12.330260 28427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:12.330569 28421 server_base.cc:1034] running on GCE node
I20260806 00:58:12.330911 28421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:12.331218 28421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:12.332403 28421 hybrid_clock.cc:648] HybridClock initialized: now 1785977892332374 us; error 43 us; skew 500 ppm
Aug 06 00:58:12 dist-test-slave-dhph krb5kdc[28168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977892, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:12.335911 28421 init.cc:377] Logged in from keytab as kudu/127.23.242.2@KRBTEST.COM (short username kudu)
I20260806 00:58:12.336447 28421 webserver.cc:460] Webserver started at http://127.23.242.2:37371/ using document root <none> and password file <none>
I20260806 00:58:12.336726 28421 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:12.336802 28421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:12.336974 28421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:12.338068 28421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-1/data/instance:
uuid: "46718badc5544bc09d8a1654698b1133"
format_stamp: "Formatted at 2026-08-06 00:58:12 on dist-test-slave-dhph"
server_key: "54819d9efbf229ce224f4a91643785a2"
server_key_iv: "86ff48202a9532aa841c690159c7076b"
server_key_version: "encryptionkey@0"
I20260806 00:58:12.338451 28421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance:
uuid: "46718badc5544bc09d8a1654698b1133"
format_stamp: "Formatted at 2026-08-06 00:58:12 on dist-test-slave-dhph"
server_key: "54819d9efbf229ce224f4a91643785a2"
server_key_iv: "86ff48202a9532aa841c690159c7076b"
server_key_version: "encryptionkey@0"
I20260806 00:58:12.340508 28421 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260806 00:58:12.341619 28436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:12.342226 28421 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:12.342365 28421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "46718badc5544bc09d8a1654698b1133"
format_stamp: "Formatted at 2026-08-06 00:58:12 on dist-test-slave-dhph"
server_key: "54819d9efbf229ce224f4a91643785a2"
server_key_iv: "86ff48202a9532aa841c690159c7076b"
server_key_version: "encryptionkey@0"
I20260806 00:58:12.342466 28421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:12.351426 28421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:12.353941 28421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:12.354183 28421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:12.354482 28421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:12.354960 28421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:12.355014 28421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:12.355144 28421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:12.355222 28421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:12.361943 28421 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:34747
I20260806 00:58:12.362071 28555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:34747 every 8 connection(s)
I20260806 00:58:12.362406 28421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 00:58:12.364696 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 28421
I20260806 00:58:12.364934 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance
I20260806 00:58:12.365182 24520 external_mini_cluster.cc:1428] Setting key 7eabb7b4d1d803e4086560bb4e1daf88
Aug 06 00:58:12 dist-test-slave-dhph krb5kdc[28168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977892, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:12.373970 28258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:12.363358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:36511 (local address 127.23.242.62:45273)
0806 00:58:12.363456 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:12.363459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:12.363709 (+ 250us) server_negotiation.cc:407] Connection header received
0806 00:58:12.364658 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:12.364660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:12.364672 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:12.364731 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:12.365618 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:12.366329 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:12.367570 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:12.367899 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:12.367974 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:12.370764 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:12.370766 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:12.370766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:12.370774 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:12.372426 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:12.372905 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:12.372906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:12.372907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:12.372974 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:12.373246 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:12.373247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:12.373247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:12.373381 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:12.373462 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:12.373778 (+ 316us) server_negotiation.cc:299] Negotiation successful
0806 00:58:12.373874 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 00:58:12.374168 28558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:12.363326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45273 (local address 127.23.242.2:36511)
0806 00:58:12.363519 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:12.363563 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:12.364531 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:12.364808 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:12.364812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:12.365205 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:12.365492 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:12.365499 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:12.366490 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:12.366492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:12.367419 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:12.367422 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:12.368021 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:12.368023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:12.368179 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:12.368884 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:12.368894 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:12.370655 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:12.372545 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:12.372554 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:12.372557 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:12.372819 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:12.373046 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:12.373047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:12.373048 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:12.373172 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:12.373453 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:12.373461 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:12.373630 (+ 169us) client_negotiation.cc:769] Sending connection context
0806 00:58:12.373743 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 00:58:12.373917 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":116,"threads_started":1}
I20260806 00:58:12.374804 28556 heartbeater.cc:347] Connected to a master server at 127.23.242.62:45273
I20260806 00:58:12.374930 28556 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:12.375123 28556 heartbeater.cc:511] Master 127.23.242.62:45273 requested a full tablet report, sending...
I20260806 00:58:12.375589 28216 ts_manager.cc:194] Registered new tserver with Master: 46718badc5544bc09d8a1654698b1133 (127.23.242.2:34747)
WARNING: no policy specified for kudu/127.23.242.3@KRBTEST.COM; defaulting to no policy
I20260806 00:58:12.376139 28216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.2@KRBTEST.COM'} at 127.23.242.2:36511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
I20260806 00:58:12.398892 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:0
--local_ip_for_outbound_sockets=127.23.242.3
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:45273
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:41071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:12.496990 28562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:12.497212 28562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:12.497239 28562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:12.498878 28562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:12.498936 28562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:12.498967 28562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 00:58:12.500555 28562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:45273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:12.500787 28562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:12.500982 28562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:12.506913 28568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:12.506932 28570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:12.506913 28567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:12.506980 28562 server_base.cc:1034] running on GCE node
I20260806 00:58:12.507316 28562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:12.507623 28562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:12.508826 28562 hybrid_clock.cc:648] HybridClock initialized: now 1785977892508797 us; error 46 us; skew 500 ppm
Aug 06 00:58:12 dist-test-slave-dhph krb5kdc[28168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977892, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:12.511885 28562 init.cc:377] Logged in from keytab as kudu/127.23.242.3@KRBTEST.COM (short username kudu)
I20260806 00:58:12.512714 28562 webserver.cc:460] Webserver started at http://127.23.242.3:44953/ using document root <none> and password file <none>
I20260806 00:58:12.513008 28562 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:12.513092 28562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:12.513276 28562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:12.514468 28562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-2/data/instance:
uuid: "2649edc746ca4cfb9c3f95a707863078"
format_stamp: "Formatted at 2026-08-06 00:58:12 on dist-test-slave-dhph"
server_key: "06916a4b285cd54574d9a3dd257cb74d"
server_key_iv: "b41b194995b4e46df243ad2a76db0ead"
server_key_version: "encryptionkey@0"
I20260806 00:58:12.514958 28562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance:
uuid: "2649edc746ca4cfb9c3f95a707863078"
format_stamp: "Formatted at 2026-08-06 00:58:12 on dist-test-slave-dhph"
server_key: "06916a4b285cd54574d9a3dd257cb74d"
server_key_iv: "b41b194995b4e46df243ad2a76db0ead"
server_key_version: "encryptionkey@0"
I20260806 00:58:12.516585 28562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 00:58:12.517594 28577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:12.518074 28562 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:12.518204 28562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "2649edc746ca4cfb9c3f95a707863078"
format_stamp: "Formatted at 2026-08-06 00:58:12 on dist-test-slave-dhph"
server_key: "06916a4b285cd54574d9a3dd257cb74d"
server_key_iv: "b41b194995b4e46df243ad2a76db0ead"
server_key_version: "encryptionkey@0"
I20260806 00:58:12.518289 28562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:12.536355 28562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:12.538946 28562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:12.539076 28562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:12.539281 28562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:12.539633 28562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:12.539672 28562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:12.539703 28562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:12.539716 28562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:12.545863 28562 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:36947
I20260806 00:58:12.545995 28696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:36947 every 8 connection(s)
I20260806 00:58:12.546511 28562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
Aug 06 00:58:12 dist-test-slave-dhph krb5kdc[28168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977892, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:12.555847 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 28562
I20260806 00:58:12.555941 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance
I20260806 00:58:12.556170 24520 external_mini_cluster.cc:1428] Setting key 2cbb40610276ff6f5ef389f70f569d67
I20260806 00:58:12.559191 28258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:12.547502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:49701 (local address 127.23.242.62:45273)
0806 00:58:12.547607 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:12.547609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:12.547833 (+ 224us) server_negotiation.cc:407] Connection header received
0806 00:58:12.549090 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:12.549091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:12.549113 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:12.549172 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:12.549980 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:12.550563 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:12.551682 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:12.551888 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:12.551957 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:12.555034 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:12.555036 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:12.555036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:12.555044 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:12.557355 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:12.557878 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:12.557879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:12.557879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:12.557939 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:12.558366 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:12.558367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:12.558367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:12.558501 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:12.558565 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:12.559035 (+ 470us) server_negotiation.cc:299] Negotiation successful
0806 00:58:12.559121 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 00:58:12.559444 28699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:12.547450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45273 (local address 127.23.242.3:49701)
0806 00:58:12.547639 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:12.547726 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:12.548939 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:12.549345 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:12.549350 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:12.549647 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:12.549867 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:12.549871 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:12.550756 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:12.550757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:12.551580 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:12.551582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:12.552071 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:12.552073 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:12.552274 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:12.553073 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:12.553083 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:12.554910 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:12.557445 (+ 2535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:12.557456 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:12.557458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:12.557791 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:12.558046 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:12.558047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:12.558047 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:12.558173 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:12.558689 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:12.558696 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:12.558842 (+ 146us) client_negotiation.cc:769] Sending connection context
0806 00:58:12.558955 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 00:58:12.559121 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":112,"threads_started":1}
I20260806 00:58:12.560364 28697 heartbeater.cc:347] Connected to a master server at 127.23.242.62:45273
I20260806 00:58:12.560766 28697 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:12.561120 28697 heartbeater.cc:511] Master 127.23.242.62:45273 requested a full tablet report, sending...
I20260806 00:58:12.561877 28216 ts_manager.cc:194] Registered new tserver with Master: 2649edc746ca4cfb9c3f95a707863078 (127.23.242.3:36947)
I20260806 00:58:12.562309 28216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.3@KRBTEST.COM'} at 127.23.242.3:49701
I20260806 00:58:12.569170 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 00:58:12 dist-test-slave-dhph krb5kdc[28168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 00:58:12 dist-test-slave-dhph krb5kdc[28168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.1@KRBTEST.COM
I20260806 00:58:12.597038 28704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:12.587842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40564 (local address 127.23.242.1:34039)
0806 00:58:12.588039 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:12.588044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:12.588055 (+ 11us) server_negotiation.cc:407] Connection header received
0806 00:58:12.588083 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:12.588085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:12.588100 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:12.588216 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:12.588869 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:12.589598 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:12.590355 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:12.590800 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:12.590902 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:12.593413 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:12.593422 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:12.593422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:12.593430 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:12.595140 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:12.595570 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:12.595574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:12.595575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:12.595647 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:12.596029 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:12.596030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:12.596030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:12.596236 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:12.596402 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:12.596763 (+ 361us) server_negotiation.cc:299] Negotiation successful
0806 00:58:12.596874 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":87,"threads_started":1}
Aug 06 00:58:12 dist-test-slave-dhph krb5kdc[28168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:12.608481 28258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:12.598758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54620 (local address 127.23.242.62:45273)
0806 00:58:12.598811 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:12.598813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:12.599083 (+ 270us) server_negotiation.cc:407] Connection header received
0806 00:58:12.599143 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:12.599145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:12.599169 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:12.599300 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:12.600138 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:12.601207 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:12.602352 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:12.602642 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:12.602744 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:12.605462 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:12.605464 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:12.605465 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:12.605482 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:12.607224 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:12.607608 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:12.607609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:12.607609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:12.607674 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:12.607925 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:12.607926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:12.607926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:12.608102 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:12.608193 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:12.608331 (+ 138us) server_negotiation.cc:299] Negotiation successful
0806 00:58:12.608408 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260806 00:58:12.608763 28216 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54620
I20260806 00:58:12.609138 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 28281
I20260806 00:58:12.615551 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 28421
I20260806 00:58:12.624353 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 28562
I20260806 00:58:12.630565 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 28181
2026-08-06T00:58:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2137 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 00:58:12 dist-test-slave-dhph krb5kdc[28709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 00:58:12 dist-test-slave-dhph krb5kdc[28709](info): set up 2 sockets
Aug 06 00:58:12 dist-test-slave-dhph krb5kdc[28709](info): commencing operation
krb5kdc: starting...
W20260806 00:58:13.778403 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 00:58:13 dist-test-slave-dhph krb5kdc[28709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T00:58:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T00:58:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 00:58:13.943959 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:46825
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:46825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:14.039318 28722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:14.039636 28722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:14.039705 28722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:14.041225 28722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 00:58:14.041329 28722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:14.041359 28722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:14.041395 28722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 00:58:14.041425 28722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 00:58:14.043291 28722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:46825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:46825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:14.043633 28722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:14.043911 28722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:14.049759 28730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:14.049769 28727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:14.049976 28722 server_base.cc:1034] running on GCE node
W20260806 00:58:14.049786 28728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:14.050410 28722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:14.050786 28722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:14.051985 28722 hybrid_clock.cc:648] HybridClock initialized: now 1785977894051971 us; error 33 us; skew 500 ppm
Aug 06 00:58:14 dist-test-slave-dhph krb5kdc[28709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977894, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:14.061211 28722 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 00:58:14.062175 28722 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:14.062255 28722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:14.062469 28722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:14.063763 28722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "4a009b4623874890bc2955e4b18d472d"
format_stamp: "Formatted at 2026-08-06 00:58:14 on dist-test-slave-dhph"
server_key: "f6048dacff4a3bb27e5a72885db06fca"
server_key_iv: "27bf328e19c3cb643eb30b52408d9f79"
server_key_version: "encryptionkey@0"
I20260806 00:58:14.064792 28722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "4a009b4623874890bc2955e4b18d472d"
format_stamp: "Formatted at 2026-08-06 00:58:14 on dist-test-slave-dhph"
server_key: "f6048dacff4a3bb27e5a72885db06fca"
server_key_iv: "27bf328e19c3cb643eb30b52408d9f79"
server_key_version: "encryptionkey@0"
I20260806 00:58:14.066447 28722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 00:58:14.067584 28736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:14.067983 28722 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:14.068079 28722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "4a009b4623874890bc2955e4b18d472d"
format_stamp: "Formatted at 2026-08-06 00:58:14 on dist-test-slave-dhph"
server_key: "f6048dacff4a3bb27e5a72885db06fca"
server_key_iv: "27bf328e19c3cb643eb30b52408d9f79"
server_key_version: "encryptionkey@0"
I20260806 00:58:14.068148 28722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:14.083177 28722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:14.086089 28722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:14.086226 28722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:14.092146 28722 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:46825
I20260806 00:58:14.092185 28790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:46825 every 8 connection(s)
I20260806 00:58:14.092499 28722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 00:58:14.093667 28791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:14.096267 28791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a009b4623874890bc2955e4b18d472d: Bootstrap starting.
I20260806 00:58:14.097198 28791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a009b4623874890bc2955e4b18d472d: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:14.097500 28791 log.cc:824] T 00000000000000000000000000000000 P 4a009b4623874890bc2955e4b18d472d: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:14.098268 28791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a009b4623874890bc2955e4b18d472d: No bootstrap required, opened a new log
I20260806 00:58:14.100111 28791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a009b4623874890bc2955e4b18d472d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a009b4623874890bc2955e4b18d472d" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 46825 } }
I20260806 00:58:14.100268 28791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a009b4623874890bc2955e4b18d472d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:14.100294 28791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a009b4623874890bc2955e4b18d472d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a009b4623874890bc2955e4b18d472d, State: Initialized, Role: FOLLOWER
I20260806 00:58:14.100409 28791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a009b4623874890bc2955e4b18d472d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a009b4623874890bc2955e4b18d472d" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 46825 } }
I20260806 00:58:14.100497 28791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a009b4623874890bc2955e4b18d472d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 00:58:14.100538 28791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a009b4623874890bc2955e4b18d472d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 00:58:14.100626 28791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a009b4623874890bc2955e4b18d472d [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:14.101470 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 28722
I20260806 00:58:14.101786 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 00:58:14.102033 24520 external_mini_cluster.cc:1428] Setting key dc2ea786d5601198547058a2779a45e0
I20260806 00:58:14.102034 28791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a009b4623874890bc2955e4b18d472d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a009b4623874890bc2955e4b18d472d" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 46825 } }
I20260806 00:58:14.102156 28791 leader_election.cc:302] T 00000000000000000000000000000000 P 4a009b4623874890bc2955e4b18d472d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a009b4623874890bc2955e4b18d472d; no voters:
I20260806 00:58:14.102826 28791 leader_election.cc:288] T 00000000000000000000000000000000 P 4a009b4623874890bc2955e4b18d472d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 00:58:14.102905 28795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a009b4623874890bc2955e4b18d472d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:14.103055 28795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a009b4623874890bc2955e4b18d472d [term 1 LEADER]: Becoming Leader. State: Replica: 4a009b4623874890bc2955e4b18d472d, State: Running, Role: LEADER
I20260806 00:58:14.103154 28791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a009b4623874890bc2955e4b18d472d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 00:58:14.103191 28795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a009b4623874890bc2955e4b18d472d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a009b4623874890bc2955e4b18d472d" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 46825 } }
I20260806 00:58:14.103690 28797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a009b4623874890bc2955e4b18d472d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a009b4623874890bc2955e4b18d472d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a009b4623874890bc2955e4b18d472d" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 46825 } } }
I20260806 00:58:14.103806 28797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a009b4623874890bc2955e4b18d472d [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:14.103690 28798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a009b4623874890bc2955e4b18d472d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a009b4623874890bc2955e4b18d472d. Latest consensus state: current_term: 1 leader_uuid: "4a009b4623874890bc2955e4b18d472d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a009b4623874890bc2955e4b18d472d" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 46825 } } }
I20260806 00:58:14.104012 28798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a009b4623874890bc2955e4b18d472d [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:14.104204 28800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 00:58:14.104862 28800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 00:58:14.106937 28800 catalog_manager.cc:1347] Generated new cluster ID: 7d747b9316d645eba26a16bbbd421e15
I20260806 00:58:14.107002 28800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 00:58:14 dist-test-slave-dhph krb5kdc[28709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:14.115886 28796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:14.102775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41600 (local address 127.23.242.62:46825)
0806 00:58:14.103172 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:14.103193 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:14.103348 (+ 155us) server_negotiation.cc:407] Connection header received
0806 00:58:14.103484 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:14.103492 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:14.103571 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:14.103731 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:14.104638 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.106396 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.107581 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.108143 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.108321 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:14.111112 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:14.111117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:14.111117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:14.111129 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:14.113687 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:14.114411 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:14.114414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:14.114415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:14.114527 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:14.114895 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:14.114896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:14.114896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:14.115180 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:14.115375 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:14.115468 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 00:58:14.115651 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":121,"threads_started":1}
I20260806 00:58:14.129889 28800 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 00:58:14.130795 28800 catalog_manager.cc:1504] Loading token signing keys...
I20260806 00:58:14.146653 28800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a009b4623874890bc2955e4b18d472d: Generated new TSK 0
I20260806 00:58:14.146909 28800 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.242.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
I20260806 00:58:14.183519 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:0
--local_ip_for_outbound_sockets=127.23.242.1
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:46825
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:14.276834 28819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:14.277109 28819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:14.277158 28819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:14.279561 28819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:14.279670 28819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:14.279740 28819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 00:58:14.281491 28819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:46825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:14.281790 28819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:14.282047 28819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:14.287966 28825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:14.287976 28827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:14.287981 28824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:14.288275 28819 server_base.cc:1034] running on GCE node
I20260806 00:58:14.288542 28819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:14.288828 28819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:14.290081 28819 hybrid_clock.cc:648] HybridClock initialized: now 1785977894290225 us; error 222 us; skew 500 ppm
Aug 06 00:58:14 dist-test-slave-dhph krb5kdc[28709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977894, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:14.293645 28819 init.cc:377] Logged in from keytab as kudu/127.23.242.1@KRBTEST.COM (short username kudu)
I20260806 00:58:14.294046 28819 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:14.294132 28819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:14.294296 28819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:14.295506 28819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-0/data/instance:
uuid: "d6fabc1d64604d95a547425099ec0b51"
format_stamp: "Formatted at 2026-08-06 00:58:14 on dist-test-slave-dhph"
server_key: "a008519a2d03bfd88c6d783a68419195"
server_key_iv: "fa8e6f18606f149607c4ae3c68e1bf85"
server_key_version: "encryptionkey@0"
I20260806 00:58:14.295922 28819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance:
uuid: "d6fabc1d64604d95a547425099ec0b51"
format_stamp: "Formatted at 2026-08-06 00:58:14 on dist-test-slave-dhph"
server_key: "a008519a2d03bfd88c6d783a68419195"
server_key_iv: "fa8e6f18606f149607c4ae3c68e1bf85"
server_key_version: "encryptionkey@0"
I20260806 00:58:14.297544 28819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 00:58:14.298538 28833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:14.298919 28819 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:14.299048 28819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "d6fabc1d64604d95a547425099ec0b51"
format_stamp: "Formatted at 2026-08-06 00:58:14 on dist-test-slave-dhph"
server_key: "a008519a2d03bfd88c6d783a68419195"
server_key_iv: "fa8e6f18606f149607c4ae3c68e1bf85"
server_key_version: "encryptionkey@0"
I20260806 00:58:14.299149 28819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:14.313596 28819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:14.316923 28819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:14.317126 28819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:14.317395 28819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:14.317739 28819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:14.317795 28819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:14.317836 28819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:14.317880 28819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:14.324651 28819 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:40507
I20260806 00:58:14.324698 28951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:40507 every 8 connection(s)
I20260806 00:58:14.324999 28819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 00:58:14.330973 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 28819
I20260806 00:58:14.331123 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance
I20260806 00:58:14.331334 24520 external_mini_cluster.cc:1428] Setting key 8a227bb0072995f2a6475210426bbbbf
Aug 06 00:58:14 dist-test-slave-dhph krb5kdc[28709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977894, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:14.336218 28796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:14.325768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:41127 (local address 127.23.242.62:46825)
0806 00:58:14.325880 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:14.325882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:14.326077 (+ 195us) server_negotiation.cc:407] Connection header received
0806 00:58:14.326726 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:14.326727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:14.326738 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:14.326805 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:14.327921 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.328439 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.329486 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.329678 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.329779 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:14.332737 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:14.332739 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:14.332740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:14.332750 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:14.334702 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:14.335132 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:14.335133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:14.335133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:14.335198 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:14.335445 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:14.335446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:14.335447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:14.335617 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:14.335700 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:14.335985 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 00:58:14.336082 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 00:58:14.336226 28954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:14.325736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:46825 (local address 127.23.242.1:41127)
0806 00:58:14.325921 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:14.325966 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:14.326467 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:14.326925 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:14.326928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:14.327223 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:14.327808 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:14.327814 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.328548 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.328549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:14.329406 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:14.329409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.329795 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.329796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:14.329930 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:14.330661 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:14.330673 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:14.332619 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:14.334798 (+ 2179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:14.334812 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:14.334813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:14.335050 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:14.335272 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:14.335272 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:14.335273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:14.335376 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:14.335703 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:14.335708 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:14.335836 (+ 128us) client_negotiation.cc:769] Sending connection context
0806 00:58:14.335923 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 00:58:14.336051 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":91,"threads_started":1}
I20260806 00:58:14.336822 28952 heartbeater.cc:347] Connected to a master server at 127.23.242.62:46825
I20260806 00:58:14.336951 28952 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:14.337113 28952 heartbeater.cc:511] Master 127.23.242.62:46825 requested a full tablet report, sending...
I20260806 00:58:14.337895 28753 ts_manager.cc:194] Registered new tserver with Master: d6fabc1d64604d95a547425099ec0b51 (127.23.242.1:40507)
I20260806 00:58:14.339030 28753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.1@KRBTEST.COM'} at 127.23.242.1:41127
WARNING: no policy specified for kudu/127.23.242.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
I20260806 00:58:14.368577 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:0
--local_ip_for_outbound_sockets=127.23.242.2
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:46825
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:14.461167 28958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:14.461395 28958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:14.461421 28958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:14.462563 28958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:14.462680 28958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:14.462716 28958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 00:58:14.464221 28958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:46825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:14.464442 28958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:14.464699 28958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:14.470112 28964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:14.470112 28966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:14.470232 28963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:14.470520 28958 server_base.cc:1034] running on GCE node
I20260806 00:58:14.470862 28958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:14.471158 28958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:14.472395 28958 hybrid_clock.cc:648] HybridClock initialized: now 1785977894472334 us; error 72 us; skew 500 ppm
Aug 06 00:58:14 dist-test-slave-dhph krb5kdc[28709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977894, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:14.476013 28958 init.cc:377] Logged in from keytab as kudu/127.23.242.2@KRBTEST.COM (short username kudu)
I20260806 00:58:14.476387 28958 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:14.476440 28958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:14.476548 28958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:14.477815 28958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-1/data/instance:
uuid: "943d57d3c4474244bc06ca85d1fbf5f2"
format_stamp: "Formatted at 2026-08-06 00:58:14 on dist-test-slave-dhph"
server_key: "b2709378d4eae3fe9025469c2c8a57c0"
server_key_iv: "931b8b8402443eb6a50336388dce903d"
server_key_version: "encryptionkey@0"
I20260806 00:58:14.478237 28958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance:
uuid: "943d57d3c4474244bc06ca85d1fbf5f2"
format_stamp: "Formatted at 2026-08-06 00:58:14 on dist-test-slave-dhph"
server_key: "b2709378d4eae3fe9025469c2c8a57c0"
server_key_iv: "931b8b8402443eb6a50336388dce903d"
server_key_version: "encryptionkey@0"
I20260806 00:58:14.479943 28958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260806 00:58:14.481084 28972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:14.481716 28958 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:14.481822 28958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "943d57d3c4474244bc06ca85d1fbf5f2"
format_stamp: "Formatted at 2026-08-06 00:58:14 on dist-test-slave-dhph"
server_key: "b2709378d4eae3fe9025469c2c8a57c0"
server_key_iv: "931b8b8402443eb6a50336388dce903d"
server_key_version: "encryptionkey@0"
I20260806 00:58:14.481953 28958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:14.490924 28958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:14.493623 28958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:14.493777 28958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:14.494190 28958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:14.494947 28958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:14.495021 28958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:14.495075 28958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:14.495112 28958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:14.504634 28958 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:41997
I20260806 00:58:14.504709 29090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:41997 every 8 connection(s)
I20260806 00:58:14.505064 28958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 00:58:14.505470 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 28958
I20260806 00:58:14.505591 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance
I20260806 00:58:14.505808 24520 external_mini_cluster.cc:1428] Setting key 985ab952fec0c9d4ba0f6cb606a07dea
Aug 06 00:58:14 dist-test-slave-dhph krb5kdc[28709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977894, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.242.3@KRBTEST.COM; defaulting to no policy
I20260806 00:58:14.519734 28796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:14.505987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:33829 (local address 127.23.242.62:46825)
0806 00:58:14.506045 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:14.506048 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:14.506367 (+ 319us) server_negotiation.cc:407] Connection header received
0806 00:58:14.507072 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:14.507073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:14.507090 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:14.507162 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:14.508809 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.509625 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.511506 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.511955 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.512080 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:14.515485 (+ 3405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:14.515487 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:14.515487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:14.515498 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:14.517824 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:14.518516 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:14.518518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:14.518518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:14.518617 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:14.519001 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:14.519002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:14.519003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:14.519153 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:14.519215 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:14.519570 (+ 355us) server_negotiation.cc:299] Negotiation successful
0806 00:58:14.519656 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260806 00:58:14.519968 29093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:14.505980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:46825 (local address 127.23.242.2:33829)
0806 00:58:14.506190 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:14.506242 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:14.506934 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:14.507249 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:14.507253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:14.507933 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:14.508677 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:14.508685 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.509744 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.509746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:14.511369 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:14.511374 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.512095 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.512097 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:14.512239 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:14.512898 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:14.512906 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:14.514962 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:14.517954 (+ 2992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:14.517964 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:14.517967 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:14.518391 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:14.518715 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:14.518716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:14.518716 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:14.518831 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:14.519256 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:14.519268 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:14.519406 (+ 138us) client_negotiation.cc:769] Sending connection context
0806 00:58:14.519496 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 00:58:14.519639 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":104,"threads_started":1}
I20260806 00:58:14.521128 29091 heartbeater.cc:347] Connected to a master server at 127.23.242.62:46825
I20260806 00:58:14.521266 29091 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:14.521399 29091 heartbeater.cc:511] Master 127.23.242.62:46825 requested a full tablet report, sending...
I20260806 00:58:14.521855 28753 ts_manager.cc:194] Registered new tserver with Master: 943d57d3c4474244bc06ca85d1fbf5f2 (127.23.242.2:41997)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.3@KRBTEST.COM" created.
I20260806 00:58:14.522624 28753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.2@KRBTEST.COM'} at 127.23.242.2:33829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
I20260806 00:58:14.548960 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:0
--local_ip_for_outbound_sockets=127.23.242.3
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:46825
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:14.650735 29096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:14.650957 29096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:14.650987 29096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:14.652269 29096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:14.652323 29096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:14.652354 29096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 00:58:14.653986 29096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:46825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:14.654263 29096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:14.654538 29096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:14.660893 29101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:14.660893 29104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:14.660971 29102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:14.661846 29096 server_base.cc:1034] running on GCE node
I20260806 00:58:14.662098 29096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:14.662360 29096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:14.663589 29096 hybrid_clock.cc:648] HybridClock initialized: now 1785977894663530 us; error 77 us; skew 500 ppm
Aug 06 00:58:14 dist-test-slave-dhph krb5kdc[28709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977894, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:14.666857 29096 init.cc:377] Logged in from keytab as kudu/127.23.242.3@KRBTEST.COM (short username kudu)
I20260806 00:58:14.667416 29096 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:14.667505 29096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:14.667690 29096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:14.668895 29096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-2/data/instance:
uuid: "93409298381048ea8370719afb7e9bdc"
format_stamp: "Formatted at 2026-08-06 00:58:14 on dist-test-slave-dhph"
server_key: "930118f95544ce2ec28ea9e471ab20ac"
server_key_iv: "0560408efb190d206f9290088d4511dc"
server_key_version: "encryptionkey@0"
I20260806 00:58:14.669350 29096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance:
uuid: "93409298381048ea8370719afb7e9bdc"
format_stamp: "Formatted at 2026-08-06 00:58:14 on dist-test-slave-dhph"
server_key: "930118f95544ce2ec28ea9e471ab20ac"
server_key_iv: "0560408efb190d206f9290088d4511dc"
server_key_version: "encryptionkey@0"
I20260806 00:58:14.671175 29096 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260806 00:58:14.672297 29110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:14.672562 29096 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:14.672675 29096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "93409298381048ea8370719afb7e9bdc"
format_stamp: "Formatted at 2026-08-06 00:58:14 on dist-test-slave-dhph"
server_key: "930118f95544ce2ec28ea9e471ab20ac"
server_key_iv: "0560408efb190d206f9290088d4511dc"
server_key_version: "encryptionkey@0"
I20260806 00:58:14.672803 29096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:14.689797 29096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:14.693066 29096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:14.693197 29096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:14.693473 29096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:14.693997 29096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:14.694046 29096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:14.694072 29096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:14.694084 29096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:14.701540 29096 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:38773
I20260806 00:58:14.701936 29096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
I20260806 00:58:14.701948 29228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:38773 every 8 connection(s)
I20260806 00:58:14.706843 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 29096
I20260806 00:58:14.706947 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance
I20260806 00:58:14.707142 24520 external_mini_cluster.cc:1428] Setting key b92b32d37f6ee404e8a483ce5b810a86
Aug 06 00:58:14 dist-test-slave-dhph krb5kdc[28709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977894, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:14.715556 28796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:14.702951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:55819 (local address 127.23.242.62:46825)
0806 00:58:14.702998 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:14.703000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:14.703460 (+ 460us) server_negotiation.cc:407] Connection header received
0806 00:58:14.704384 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:14.704385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:14.704396 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:14.704458 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:14.705779 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.706314 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.708033 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.708320 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.708395 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:14.711103 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:14.711107 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:14.711107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:14.711120 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:14.713568 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:14.714318 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:14.714319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:14.714319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:14.714387 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:14.714794 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:14.714795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:14.714795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:14.714927 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:14.714992 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:14.715393 (+ 401us) server_negotiation.cc:299] Negotiation successful
0806 00:58:14.715478 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 00:58:14.715909 29231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:14.702981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:46825 (local address 127.23.242.3:55819)
0806 00:58:14.703305 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:14.703349 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:14.704246 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:14.704587 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:14.704590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:14.704974 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:14.705597 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:14.705603 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.706425 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.706426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:14.707941 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:14.707945 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.708421 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.708422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:14.708538 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:14.709166 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:14.709178 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:14.710919 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:14.713727 (+ 2808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:14.713736 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:14.713738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:14.714123 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:14.714490 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:14.714491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:14.714491 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:14.714682 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:14.715054 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:14.715061 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:14.715241 (+ 180us) client_negotiation.cc:769] Sending connection context
0806 00:58:14.715349 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 00:58:14.715496 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":75,"threads_started":1}
I20260806 00:58:14.716712 29229 heartbeater.cc:347] Connected to a master server at 127.23.242.62:46825
I20260806 00:58:14.716816 29229 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:14.717046 29229 heartbeater.cc:511] Master 127.23.242.62:46825 requested a full tablet report, sending...
I20260806 00:58:14.717528 28753 ts_manager.cc:194] Registered new tserver with Master: 93409298381048ea8370719afb7e9bdc (127.23.242.3:38773)
I20260806 00:58:14.717985 28753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.3@KRBTEST.COM'} at 127.23.242.3:55819
I20260806 00:58:14.720623 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 00:58:14.730307 28796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:14.722091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41610 (local address 127.23.242.62:46825)
0806 00:58:14.722273 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:14.722275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:14.722366 (+ 91us) server_negotiation.cc:407] Connection header received
0806 00:58:14.722448 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:14.722449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:14.722468 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:14.722534 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:14.723311 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.723869 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.724714 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.724930 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.725021 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:14.727161 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:14.727164 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:14.727164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:14.727172 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:14.728892 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:14.729399 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:14.729400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:14.729400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:14.729460 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:14.729671 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:14.729672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:14.729672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:14.729844 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:14.729950 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:14.730113 (+ 163us) server_negotiation.cc:299] Negotiation successful
0806 00:58:14.730200 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260806 00:58:14.732826 28753 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 00:58:14.734568 28753 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 00:58:14.743399 29242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:14.738767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:41997 (local address 127.0.0.1:53984)
0806 00:58:14.738919 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:14.738929 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:14.738989 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:14.739479 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:14.739480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:14.739482 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:14.739543 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:14.739545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.740874 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.740875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:14.742110 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:14.742114 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.742997 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.742998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:14.743085 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:14.743117 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 00:58:14.743225 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 00:58:14.743294 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
I20260806 00:58:14.743690 29244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:14.738896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53984 (local address 127.23.242.2:41997)
0806 00:58:14.739272 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:14.739276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:14.739286 (+ 10us) server_negotiation.cc:407] Connection header received
0806 00:58:14.739333 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:14.739335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:14.739352 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:14.739444 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:14.739623 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.740759 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.742223 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.742894 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.742994 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:14.743061 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:14.743555 (+ 494us) server_negotiation.cc:299] Negotiation successful
0806 00:58:14.743607 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":94,"threads_started":1}
I20260806 00:58:14.743888 29240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:14.737951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:40507 (local address 127.0.0.1:45000)
0806 00:58:14.738692 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:14.738743 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:14.739098 (+ 355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:14.739340 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:14.739344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:14.739346 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:14.739436 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:14.739439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.740633 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.740634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:14.741999 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:14.742008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.743587 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.743589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:14.743670 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:14.743675 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 00:58:14.743740 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 00:58:14.743803 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":68,"threads_started":1}
I20260806 00:58:14.744205 29245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:14.738999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:38773 (local address 127.0.0.1:58902)
0806 00:58:14.739579 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:14.739592 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:14.739657 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:14.739861 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:14.739861 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:14.739863 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:14.739905 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:14.739907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.741202 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.741203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:14.742330 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:14.742332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.743837 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.743838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:14.743902 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:14.743907 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 00:58:14.744093 (+ 186us) client_negotiation.cc:240] Negotiation successful
0806 00:58:14.744137 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":74,"threads_started":1}
I20260806 00:58:14.744230 29241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:14.738085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45000 (local address 127.23.242.1:40507)
0806 00:58:14.738305 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:14.738309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:14.739063 (+ 754us) server_negotiation.cc:407] Connection header received
0806 00:58:14.739233 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:14.739236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:14.739250 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:14.739312 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:14.739538 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.740521 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.742189 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.743204 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.743412 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:14.743525 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:14.744027 (+ 502us) server_negotiation.cc:299] Negotiation successful
0806 00:58:14.744123 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"spinlock_wait_cycles":9984,"thread_start_us":140,"threads_started":1}
I20260806 00:58:14.744288 29243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:14.738811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58902 (local address 127.23.242.3:38773)
0806 00:58:14.739170 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:14.739176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:14.739653 (+ 477us) server_negotiation.cc:407] Connection header received
0806 00:58:14.739739 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:14.739741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:14.739757 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:14.739836 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:14.739970 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.741083 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.742451 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.743237 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.743435 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:14.743799 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:14.744131 (+ 332us) server_negotiation.cc:299] Negotiation successful
0806 00:58:14.744206 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":109,"threads_started":1}
I20260806 00:58:14.744412 29021 tablet_service.cc:1467] Processing CreateTablet for tablet 95e68b5b2cd44d56a59617832d78e7b2 (DEFAULT_TABLE table=test-table [id=ec6021021a894b6fa05479484ddd50a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:14.744848 29021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95e68b5b2cd44d56a59617832d78e7b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:14.745188 29160 tablet_service.cc:1467] Processing CreateTablet for tablet 95e68b5b2cd44d56a59617832d78e7b2 (DEFAULT_TABLE table=test-table [id=ec6021021a894b6fa05479484ddd50a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:14.745517 29160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95e68b5b2cd44d56a59617832d78e7b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:14.747104 28884 tablet_service.cc:1467] Processing CreateTablet for tablet 95e68b5b2cd44d56a59617832d78e7b2 (DEFAULT_TABLE table=test-table [id=ec6021021a894b6fa05479484ddd50a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:14.747478 28884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95e68b5b2cd44d56a59617832d78e7b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:14.747908 29246 tablet_bootstrap.cc:492] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2: Bootstrap starting.
I20260806 00:58:14.748667 29246 tablet_bootstrap.cc:654] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:14.748852 29247 tablet_bootstrap.cc:492] T 95e68b5b2cd44d56a59617832d78e7b2 P 93409298381048ea8370719afb7e9bdc: Bootstrap starting.
I20260806 00:58:14.749632 29246 log.cc:824] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:14.749887 29247 tablet_bootstrap.cc:654] T 95e68b5b2cd44d56a59617832d78e7b2 P 93409298381048ea8370719afb7e9bdc: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:14.750510 29247 log.cc:824] T 95e68b5b2cd44d56a59617832d78e7b2 P 93409298381048ea8370719afb7e9bdc: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:14.750696 29246 tablet_bootstrap.cc:492] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2: No bootstrap required, opened a new log
I20260806 00:58:14.750780 29246 ts_tablet_manager.cc:1397] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260806 00:58:14.751214 29250 tablet_bootstrap.cc:492] T 95e68b5b2cd44d56a59617832d78e7b2 P d6fabc1d64604d95a547425099ec0b51: Bootstrap starting.
I20260806 00:58:14.752115 29250 tablet_bootstrap.cc:654] T 95e68b5b2cd44d56a59617832d78e7b2 P d6fabc1d64604d95a547425099ec0b51: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:14.752161 29247 tablet_bootstrap.cc:492] T 95e68b5b2cd44d56a59617832d78e7b2 P 93409298381048ea8370719afb7e9bdc: No bootstrap required, opened a new log
I20260806 00:58:14.752260 29247 ts_tablet_manager.cc:1397] T 95e68b5b2cd44d56a59617832d78e7b2 P 93409298381048ea8370719afb7e9bdc: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260806 00:58:14.752782 29250 log.cc:824] T 95e68b5b2cd44d56a59617832d78e7b2 P d6fabc1d64604d95a547425099ec0b51: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:14.753919 29250 tablet_bootstrap.cc:492] T 95e68b5b2cd44d56a59617832d78e7b2 P d6fabc1d64604d95a547425099ec0b51: No bootstrap required, opened a new log
I20260806 00:58:14.753808 29246 raft_consensus.cc:348] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6fabc1d64604d95a547425099ec0b51" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 40507 } } peers { permanent_uuid: "93409298381048ea8370719afb7e9bdc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 38773 } } peers { permanent_uuid: "943d57d3c4474244bc06ca85d1fbf5f2" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41997 } }
I20260806 00:58:14.754004 29250 ts_tablet_manager.cc:1397] T 95e68b5b2cd44d56a59617832d78e7b2 P d6fabc1d64604d95a547425099ec0b51: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 00:58:14.754001 29246 raft_consensus.cc:374] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:14.754043 29246 raft_consensus.cc:729] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 943d57d3c4474244bc06ca85d1fbf5f2, State: Initialized, Role: FOLLOWER
I20260806 00:58:14.754228 29246 consensus_queue.cc:260] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6fabc1d64604d95a547425099ec0b51" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 40507 } } peers { permanent_uuid: "93409298381048ea8370719afb7e9bdc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 38773 } } peers { permanent_uuid: "943d57d3c4474244bc06ca85d1fbf5f2" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41997 } }
I20260806 00:58:14.754601 29246 ts_tablet_manager.cc:1428] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260806 00:58:14.754766 29091 heartbeater.cc:503] Master 127.23.242.62:46825 was elected leader, sending a full tablet report...
I20260806 00:58:14.755278 29247 raft_consensus.cc:348] T 95e68b5b2cd44d56a59617832d78e7b2 P 93409298381048ea8370719afb7e9bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6fabc1d64604d95a547425099ec0b51" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 40507 } } peers { permanent_uuid: "93409298381048ea8370719afb7e9bdc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 38773 } } peers { permanent_uuid: "943d57d3c4474244bc06ca85d1fbf5f2" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41997 } }
I20260806 00:58:14.755458 29247 raft_consensus.cc:374] T 95e68b5b2cd44d56a59617832d78e7b2 P 93409298381048ea8370719afb7e9bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:14.755510 29247 raft_consensus.cc:729] T 95e68b5b2cd44d56a59617832d78e7b2 P 93409298381048ea8370719afb7e9bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93409298381048ea8370719afb7e9bdc, State: Initialized, Role: FOLLOWER
W20260806 00:58:14.755761 29092 tablet.cc:2369] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 00:58:14.756023 29247 consensus_queue.cc:260] T 95e68b5b2cd44d56a59617832d78e7b2 P 93409298381048ea8370719afb7e9bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6fabc1d64604d95a547425099ec0b51" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 40507 } } peers { permanent_uuid: "93409298381048ea8370719afb7e9bdc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 38773 } } peers { permanent_uuid: "943d57d3c4474244bc06ca85d1fbf5f2" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41997 } }
I20260806 00:58:14.756420 29247 ts_tablet_manager.cc:1428] T 95e68b5b2cd44d56a59617832d78e7b2 P 93409298381048ea8370719afb7e9bdc: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260806 00:58:14.756572 29229 heartbeater.cc:503] Master 127.23.242.62:46825 was elected leader, sending a full tablet report...
I20260806 00:58:14.757594 29250 raft_consensus.cc:348] T 95e68b5b2cd44d56a59617832d78e7b2 P d6fabc1d64604d95a547425099ec0b51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6fabc1d64604d95a547425099ec0b51" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 40507 } } peers { permanent_uuid: "93409298381048ea8370719afb7e9bdc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 38773 } } peers { permanent_uuid: "943d57d3c4474244bc06ca85d1fbf5f2" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41997 } }
I20260806 00:58:14.758147 29250 raft_consensus.cc:374] T 95e68b5b2cd44d56a59617832d78e7b2 P d6fabc1d64604d95a547425099ec0b51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:14.758235 29250 raft_consensus.cc:729] T 95e68b5b2cd44d56a59617832d78e7b2 P d6fabc1d64604d95a547425099ec0b51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6fabc1d64604d95a547425099ec0b51, State: Initialized, Role: FOLLOWER
I20260806 00:58:14.758510 29250 consensus_queue.cc:260] T 95e68b5b2cd44d56a59617832d78e7b2 P d6fabc1d64604d95a547425099ec0b51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6fabc1d64604d95a547425099ec0b51" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 40507 } } peers { permanent_uuid: "93409298381048ea8370719afb7e9bdc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 38773 } } peers { permanent_uuid: "943d57d3c4474244bc06ca85d1fbf5f2" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41997 } }
I20260806 00:58:14.759099 29250 ts_tablet_manager.cc:1428] T 95e68b5b2cd44d56a59617832d78e7b2 P d6fabc1d64604d95a547425099ec0b51: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 00:58:14.759106 28952 heartbeater.cc:503] Master 127.23.242.62:46825 was elected leader, sending a full tablet report...
I20260806 00:58:14.808948 29252 raft_consensus.cc:482] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 00:58:14.809134 29252 raft_consensus.cc:504] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6fabc1d64604d95a547425099ec0b51" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 40507 } } peers { permanent_uuid: "93409298381048ea8370719afb7e9bdc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 38773 } } peers { permanent_uuid: "943d57d3c4474244bc06ca85d1fbf5f2" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41997 } }
I20260806 00:58:14.809558 29252 leader_election.cc:288] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6fabc1d64604d95a547425099ec0b51 (127.23.242.1:40507), 93409298381048ea8370719afb7e9bdc (127.23.242.3:38773)
I20260806 00:58:14.813938 29241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:14.809913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:59953 (local address 127.23.242.1:40507)
0806 00:58:14.809961 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:14.809964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:14.810096 (+ 132us) server_negotiation.cc:407] Connection header received
0806 00:58:14.810184 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:14.810185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:14.810196 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:14.810258 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:14.810435 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.811276 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.812898 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.813471 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.813558 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:14.813568 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:14.813818 (+ 250us) server_negotiation.cc:299] Negotiation successful
0806 00:58:14.813865 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 00:58:14.813938 29243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:14.809869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:49793 (local address 127.23.242.3:38773)
0806 00:58:14.809921 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:14.809924 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:14.809957 (+ 33us) server_negotiation.cc:407] Connection header received
0806 00:58:14.810031 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:14.810032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:14.810041 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:14.810124 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:14.810337 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.811203 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.812898 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.813483 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.813558 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:14.813568 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:14.813817 (+ 249us) server_negotiation.cc:299] Negotiation successful
0806 00:58:14.813865 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 00:58:14.813958 29093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:14.809785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:38773 (local address 127.23.242.2:49793)
0806 00:58:14.809890 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:14.809905 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:14.809954 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:14.810149 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:14.810151 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:14.810153 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:14.810231 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:14.810232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.811363 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.811364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:14.812764 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:14.812767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.813592 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.813594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:14.813698 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:14.813704 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 00:58:14.813787 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 00:58:14.813848 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":100}
I20260806 00:58:14.813964 29255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:14.809805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:40507 (local address 127.23.242.2:59953)
0806 00:58:14.810011 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:14.810021 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:14.810084 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:14.810279 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:14.810280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:14.810282 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:14.810348 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:14.810349 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.811371 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.811371 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:14.812764 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:14.812767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.813592 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.813594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:14.813717 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:14.813721 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 00:58:14.813787 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 00:58:14.813848 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":47,"thread_start_us":127,"threads_started":1}
I20260806 00:58:14.814422 29182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e68b5b2cd44d56a59617832d78e7b2" candidate_uuid: "943d57d3c4474244bc06ca85d1fbf5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93409298381048ea8370719afb7e9bdc" is_pre_election: true
I20260806 00:58:14.814567 28905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e68b5b2cd44d56a59617832d78e7b2" candidate_uuid: "943d57d3c4474244bc06ca85d1fbf5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6fabc1d64604d95a547425099ec0b51" is_pre_election: true
I20260806 00:58:14.814652 29182 raft_consensus.cc:2454] T 95e68b5b2cd44d56a59617832d78e7b2 P 93409298381048ea8370719afb7e9bdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 943d57d3c4474244bc06ca85d1fbf5f2 in term 0.
I20260806 00:58:14.814757 28905 raft_consensus.cc:2454] T 95e68b5b2cd44d56a59617832d78e7b2 P d6fabc1d64604d95a547425099ec0b51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 943d57d3c4474244bc06ca85d1fbf5f2 in term 0.
I20260806 00:58:14.815016 28974 leader_election.cc:302] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93409298381048ea8370719afb7e9bdc, 943d57d3c4474244bc06ca85d1fbf5f2; no voters:
I20260806 00:58:14.815227 29252 raft_consensus.cc:2781] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 00:58:14.815294 29252 raft_consensus.cc:482] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 00:58:14.815330 29252 raft_consensus.cc:3037] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:14.816099 29252 raft_consensus.cc:504] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6fabc1d64604d95a547425099ec0b51" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 40507 } } peers { permanent_uuid: "93409298381048ea8370719afb7e9bdc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 38773 } } peers { permanent_uuid: "943d57d3c4474244bc06ca85d1fbf5f2" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41997 } }
I20260806 00:58:14.816277 29252 leader_election.cc:288] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2 [CANDIDATE]: Term 1 election: Requested vote from peers d6fabc1d64604d95a547425099ec0b51 (127.23.242.1:40507), 93409298381048ea8370719afb7e9bdc (127.23.242.3:38773)
I20260806 00:58:14.816439 28905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e68b5b2cd44d56a59617832d78e7b2" candidate_uuid: "943d57d3c4474244bc06ca85d1fbf5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6fabc1d64604d95a547425099ec0b51"
I20260806 00:58:14.816538 28905 raft_consensus.cc:3037] T 95e68b5b2cd44d56a59617832d78e7b2 P d6fabc1d64604d95a547425099ec0b51 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:14.816573 29182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e68b5b2cd44d56a59617832d78e7b2" candidate_uuid: "943d57d3c4474244bc06ca85d1fbf5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93409298381048ea8370719afb7e9bdc"
I20260806 00:58:14.816671 29182 raft_consensus.cc:3037] T 95e68b5b2cd44d56a59617832d78e7b2 P 93409298381048ea8370719afb7e9bdc [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:14.817386 28905 raft_consensus.cc:2454] T 95e68b5b2cd44d56a59617832d78e7b2 P d6fabc1d64604d95a547425099ec0b51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 943d57d3c4474244bc06ca85d1fbf5f2 in term 1.
I20260806 00:58:14.817420 29182 raft_consensus.cc:2454] T 95e68b5b2cd44d56a59617832d78e7b2 P 93409298381048ea8370719afb7e9bdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 943d57d3c4474244bc06ca85d1fbf5f2 in term 1.
I20260806 00:58:14.817585 28973 leader_election.cc:302] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 943d57d3c4474244bc06ca85d1fbf5f2, d6fabc1d64604d95a547425099ec0b51; no voters:
I20260806 00:58:14.817701 29252 raft_consensus.cc:2781] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:14.817903 29252 raft_consensus.cc:686] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2 [term 1 LEADER]: Becoming Leader. State: Replica: 943d57d3c4474244bc06ca85d1fbf5f2, State: Running, Role: LEADER
I20260806 00:58:14.818018 29252 consensus_queue.cc:237] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6fabc1d64604d95a547425099ec0b51" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 40507 } } peers { permanent_uuid: "93409298381048ea8370719afb7e9bdc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 38773 } } peers { permanent_uuid: "943d57d3c4474244bc06ca85d1fbf5f2" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41997 } }
I20260806 00:58:14.818993 28753 catalog_manager.cc:5495] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 943d57d3c4474244bc06ca85d1fbf5f2 (127.23.242.2). New cstate: current_term: 1 leader_uuid: "943d57d3c4474244bc06ca85d1fbf5f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6fabc1d64604d95a547425099ec0b51" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 40507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93409298381048ea8370719afb7e9bdc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 38773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "943d57d3c4474244bc06ca85d1fbf5f2" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41997 } health_report { overall_health: HEALTHY } } }
W20260806 00:58:14.825819 28953 tablet.cc:2369] T 95e68b5b2cd44d56a59617832d78e7b2 P d6fabc1d64604d95a547425099ec0b51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 00:58:14.840152 29244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:14.837267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53996 (local address 127.23.242.2:41997)
0806 00:58:14.837353 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:14.837355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:14.837361 (+ 6us) server_negotiation.cc:407] Connection header received
0806 00:58:14.837399 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:14.837400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:14.837419 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:14.837498 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 00:58:14.837676 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.838183 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.839041 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:14.839423 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:14.839543 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:14.839766 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 00:58:14.839872 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 00:58:14.839965 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:14.840005 (+ 40us) server_negotiation.cc:299] Negotiation successful
0806 00:58:14.840026 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 00:58:14.842376 29182 raft_consensus.cc:1261] T 95e68b5b2cd44d56a59617832d78e7b2 P 93409298381048ea8370719afb7e9bdc [term 1 FOLLOWER]: Refusing update from remote peer 943d57d3c4474244bc06ca85d1fbf5f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:14.842410 28905 raft_consensus.cc:1261] T 95e68b5b2cd44d56a59617832d78e7b2 P d6fabc1d64604d95a547425099ec0b51 [term 1 FOLLOWER]: Refusing update from remote peer 943d57d3c4474244bc06ca85d1fbf5f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:14.842886 29256 consensus_queue.cc:1035] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93409298381048ea8370719afb7e9bdc" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 38773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:14.843044 29252 consensus_queue.cc:1035] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6fabc1d64604d95a547425099ec0b51" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 40507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:14.845440 29259 mvcc.cc:205] Tried to move back new op lower bound from 7315365457270886400 to 7315365457175257088. Current Snapshot: MvccSnapshot[applied={T|T < 7315365457270886400}]
I20260806 00:58:14.845602 29260 mvcc.cc:205] Tried to move back new op lower bound from 7315365457270886400 to 7315365457175257088. Current Snapshot: MvccSnapshot[applied={T|T < 7315365457270886400}]
I20260806 00:58:14.846252 29261 mvcc.cc:205] Tried to move back new op lower bound from 7315365457270886400 to 7315365457175257088. Current Snapshot: MvccSnapshot[applied={T|T < 7315365457270886400}]
I20260806 00:58:14.849406 28752 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:58:14.849507 28752 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:58:14.850400 28752 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a009b4623874890bc2955e4b18d472d: Sending DeleteTablet for 3 replicas of tablet 95e68b5b2cd44d56a59617832d78e7b2
I20260806 00:58:14.850834 29021 tablet_service.cc:1514] Processing DeleteTablet for tablet 95e68b5b2cd44d56a59617832d78e7b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:14 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:53984
I20260806 00:58:14.850970 28884 tablet_service.cc:1514] Processing DeleteTablet for tablet 95e68b5b2cd44d56a59617832d78e7b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:14 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:45000
I20260806 00:58:14.851118 29160 tablet_service.cc:1514] Processing DeleteTablet for tablet 95e68b5b2cd44d56a59617832d78e7b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:14 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:58902
I20260806 00:58:14.851380 29270 tablet_replica.cc:330] T 95e68b5b2cd44d56a59617832d78e7b2 P d6fabc1d64604d95a547425099ec0b51: stopping tablet replica
I20260806 00:58:14.851429 29268 tablet_replica.cc:330] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2: stopping tablet replica
I20260806 00:58:14.851517 29268 raft_consensus.cc:2229] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2 [term 1 LEADER]: Raft consensus shutting down.
I20260806 00:58:14.851518 29270 raft_consensus.cc:2229] T 95e68b5b2cd44d56a59617832d78e7b2 P d6fabc1d64604d95a547425099ec0b51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:14.851637 29270 raft_consensus.cc:2258] T 95e68b5b2cd44d56a59617832d78e7b2 P d6fabc1d64604d95a547425099ec0b51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:14.851650 29268 raft_consensus.cc:2258] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:14.851958 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 28819
I20260806 00:58:14.853471 29271 tablet_replica.cc:330] T 95e68b5b2cd44d56a59617832d78e7b2 P 93409298381048ea8370719afb7e9bdc: stopping tablet replica
I20260806 00:58:14.853600 29271 raft_consensus.cc:2229] T 95e68b5b2cd44d56a59617832d78e7b2 P 93409298381048ea8370719afb7e9bdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:14.853691 29271 raft_consensus.cc:2258] T 95e68b5b2cd44d56a59617832d78e7b2 P 93409298381048ea8370719afb7e9bdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:14.854102 29271 ts_tablet_manager.cc:1905] T 95e68b5b2cd44d56a59617832d78e7b2 P 93409298381048ea8370719afb7e9bdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:14.854619 29268 ts_tablet_manager.cc:1905] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:14.856401 29271 ts_tablet_manager.cc:1918] T 95e68b5b2cd44d56a59617832d78e7b2 P 93409298381048ea8370719afb7e9bdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 00:58:14.856511 29271 log.cc:1196] T 95e68b5b2cd44d56a59617832d78e7b2 P 93409298381048ea8370719afb7e9bdc: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-2/wal/wals/95e68b5b2cd44d56a59617832d78e7b2
I20260806 00:58:14.856509 29268 ts_tablet_manager.cc:1918] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 00:58:14.856578 29268 log.cc:1196] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785977869987768-24520-0/minicluster-data/ts-1/wal/wals/95e68b5b2cd44d56a59617832d78e7b2
I20260806 00:58:14.856886 29271 ts_tablet_manager.cc:1939] T 95e68b5b2cd44d56a59617832d78e7b2 P 93409298381048ea8370719afb7e9bdc: Deleting consensus metadata
I20260806 00:58:14.856895 29268 ts_tablet_manager.cc:1939] T 95e68b5b2cd44d56a59617832d78e7b2 P 943d57d3c4474244bc06ca85d1fbf5f2: Deleting consensus metadata
I20260806 00:58:14.857345 28739 catalog_manager.cc:4841] TS 943d57d3c4474244bc06ca85d1fbf5f2 (127.23.242.2:41997): tablet 95e68b5b2cd44d56a59617832d78e7b2 (table test-table [id=ec6021021a894b6fa05479484ddd50a7]) successfully deleted
I20260806 00:58:14.857414 28738 catalog_manager.cc:4841] TS 93409298381048ea8370719afb7e9bdc (127.23.242.3:38773): tablet 95e68b5b2cd44d56a59617832d78e7b2 (table test-table [id=ec6021021a894b6fa05479484ddd50a7]) successfully deleted
W20260806 00:58:14.861100 28737 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.242.1:40507 (error 108)
I20260806 00:58:14.861790 29245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:14.861638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:40507 (local address 127.0.0.1:45008)
0806 00:58:14.861705 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:14.861747 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.1:40507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 00:58:14.861812 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 28958
W20260806 00:58:14.862946 28737 catalog_manager.cc:4584] TS d6fabc1d64604d95a547425099ec0b51 (127.23.242.1:40507): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 95e68b5b2cd44d56a59617832d78e7b2: Network error: Client connection negotiation failed: client connection to 127.23.242.1:40507: connect: Connection refused (error 111)
I20260806 00:58:14.868199 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 29096
I20260806 00:58:14.873939 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 28722
2026-08-06T00:58:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2242 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T00:58:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T00:58:14Z Disabled control of system clock
I20260806 00:58:14.907876 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:33071
--webserver_interface=127.23.242.62
--webserver_port=0
--builtin_ntp_servers=127.23.242.20:41525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:33071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 00:58:15.000960 29276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:15.001243 29276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:15.001274 29276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:15.002913 29276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 00:58:15.002960 29276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:15.002974 29276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:15.002986 29276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 00:58:15.002998 29276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 00:58:15.004608 29276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:33071
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:33071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:15.004870 29276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:15.005602 29276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:15.012020 29281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:15.011943 29282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:15.011945 29284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:15.012447 29276 server_base.cc:1034] running on GCE node
I20260806 00:58:15.012737 29276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:15.013119 29276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:15.014339 29276 hybrid_clock.cc:648] HybridClock initialized: now 1785977895014295 us; error 55 us; skew 500 ppm
I20260806 00:58:15.016516 29276 webserver.cc:460] Webserver started at http://127.23.242.62:46593/ using document root <none> and password file <none>
I20260806 00:58:15.016919 29276 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:15.017014 29276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:15.017195 29276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:15.018443 29276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "6a3fb9e3147546f5a701259705f3c81d"
format_stamp: "Formatted at 2026-08-06 00:58:15 on dist-test-slave-dhph"
server_key: "0eabcfd1e0c5c37b32cb880a5e4d0ea6"
server_key_iv: "878fc294e542a1816958ce10faf9f7f6"
server_key_version: "encryptionkey@0"
I20260806 00:58:15.019065 29276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "6a3fb9e3147546f5a701259705f3c81d"
format_stamp: "Formatted at 2026-08-06 00:58:15 on dist-test-slave-dhph"
server_key: "0eabcfd1e0c5c37b32cb880a5e4d0ea6"
server_key_iv: "878fc294e542a1816958ce10faf9f7f6"
server_key_version: "encryptionkey@0"
I20260806 00:58:15.020896 29276 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260806 00:58:15.022799 29290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:15.023072 29276 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260806 00:58:15.023221 29276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "6a3fb9e3147546f5a701259705f3c81d"
format_stamp: "Formatted at 2026-08-06 00:58:15 on dist-test-slave-dhph"
server_key: "0eabcfd1e0c5c37b32cb880a5e4d0ea6"
server_key_iv: "878fc294e542a1816958ce10faf9f7f6"
server_key_version: "encryptionkey@0"
I20260806 00:58:15.023327 29276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:15.028319 29276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:15.028678 29276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:15.028852 29276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:15.033543 29276 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:33071
I20260806 00:58:15.033716 29344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:33071 every 8 connection(s)
I20260806 00:58:15.034080 29276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 00:58:15.034567 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 29276
I20260806 00:58:15.034759 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 00:58:15.034982 24520 external_mini_cluster.cc:1428] Setting key 2481e5fbcaefe95118e1a2207467248c
I20260806 00:58:15.035110 29345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:15.038657 29348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:15.036002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40988 (local address 127.23.242.62:33071)
0806 00:58:15.036292 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:15.036300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:15.036334 (+ 34us) server_negotiation.cc:407] Connection header received
0806 00:58:15.036988 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:15.036995 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:15.037041 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:15.037228 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:15.038092 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:15.038096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:15.038096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 00:58:15.038101 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:15.038152 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 00:58:15.038163 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:15.038256 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:15.038336 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 00:58:15.038401 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":137,"threads_started":1}
I20260806 00:58:15.039285 29345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a3fb9e3147546f5a701259705f3c81d: Bootstrap starting.
I20260806 00:58:15.040000 29345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a3fb9e3147546f5a701259705f3c81d: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:15.040303 29345 log.cc:824] T 00000000000000000000000000000000 P 6a3fb9e3147546f5a701259705f3c81d: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:15.041206 29345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a3fb9e3147546f5a701259705f3c81d: No bootstrap required, opened a new log
I20260806 00:58:15.042966 29345 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a3fb9e3147546f5a701259705f3c81d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3fb9e3147546f5a701259705f3c81d" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 33071 } }
I20260806 00:58:15.043123 29345 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a3fb9e3147546f5a701259705f3c81d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:15.043147 29345 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a3fb9e3147546f5a701259705f3c81d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3fb9e3147546f5a701259705f3c81d, State: Initialized, Role: FOLLOWER
I20260806 00:58:15.043269 29345 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a3fb9e3147546f5a701259705f3c81d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3fb9e3147546f5a701259705f3c81d" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 33071 } }
I20260806 00:58:15.043356 29345 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a3fb9e3147546f5a701259705f3c81d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 00:58:15.043380 29345 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a3fb9e3147546f5a701259705f3c81d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 00:58:15.043411 29345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a3fb9e3147546f5a701259705f3c81d [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:15.043983 29345 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a3fb9e3147546f5a701259705f3c81d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3fb9e3147546f5a701259705f3c81d" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 33071 } }
I20260806 00:58:15.044071 29345 leader_election.cc:302] T 00000000000000000000000000000000 P 6a3fb9e3147546f5a701259705f3c81d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a3fb9e3147546f5a701259705f3c81d; no voters:
I20260806 00:58:15.044236 29345 leader_election.cc:288] T 00000000000000000000000000000000 P 6a3fb9e3147546f5a701259705f3c81d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 00:58:15.044521 29350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a3fb9e3147546f5a701259705f3c81d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:15.044559 29345 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a3fb9e3147546f5a701259705f3c81d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 00:58:15.044660 29350 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a3fb9e3147546f5a701259705f3c81d [term 1 LEADER]: Becoming Leader. State: Replica: 6a3fb9e3147546f5a701259705f3c81d, State: Running, Role: LEADER
I20260806 00:58:15.044770 29350 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a3fb9e3147546f5a701259705f3c81d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3fb9e3147546f5a701259705f3c81d" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 33071 } }
I20260806 00:58:15.045099 29350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a3fb9e3147546f5a701259705f3c81d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a3fb9e3147546f5a701259705f3c81d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3fb9e3147546f5a701259705f3c81d" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 33071 } } }
I20260806 00:58:15.045204 29350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a3fb9e3147546f5a701259705f3c81d [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:15.045358 29350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a3fb9e3147546f5a701259705f3c81d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a3fb9e3147546f5a701259705f3c81d. Latest consensus state: current_term: 1 leader_uuid: "6a3fb9e3147546f5a701259705f3c81d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3fb9e3147546f5a701259705f3c81d" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 33071 } } }
I20260806 00:58:15.045420 29350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a3fb9e3147546f5a701259705f3c81d [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:15.045428 29360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 00:58:15.046099 29360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 00:58:15.048563 29360 catalog_manager.cc:1347] Generated new cluster ID: 7fbaadad91df4735ab7f5f27a396cc56
I20260806 00:58:15.048646 29360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 00:58:15.107604 29360 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 00:58:15.107820 29360 catalog_manager.cc:1504] Loading token signing keys...
I20260806 00:58:15.115247 29360 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a3fb9e3147546f5a701259705f3c81d: Generated new TSK 0
I20260806 00:58:15.115465 29360 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 00:58:15.119055 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:0
--local_ip_for_outbound_sockets=127.23.242.1
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:33071
--builtin_ntp_servers=127.23.242.20:41525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 00:58:15.217515 29370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:15.217783 29370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:15.217813 29370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:15.219787 29370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:15.219853 29370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:15.219888 29370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 00:58:15.222218 29370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:33071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:15.222627 29370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:15.223218 29370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 00:58:15.230371 29370 server_base.cc:1034] running on GCE node
W20260806 00:58:15.230449 29376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:15.230616 29375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:15.230485 29378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:15.230919 29370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:15.231189 29370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:15.232379 29370 hybrid_clock.cc:648] HybridClock initialized: now 1785977895232364 us; error 32 us; skew 500 ppm
I20260806 00:58:15.233915 29370 webserver.cc:460] Webserver started at http://127.23.242.1:35381/ using document root <none> and password file <none>
I20260806 00:58:15.234297 29370 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:15.234375 29370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:15.234539 29370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:15.235764 29370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-0/data/instance:
uuid: "ffdfba0e9ea44e528f8d6d55415afd40"
format_stamp: "Formatted at 2026-08-06 00:58:15 on dist-test-slave-dhph"
server_key: "a4e68a4260fed5094f408b5cc5c2d80e"
server_key_iv: "9b98c9af6e73865916d446d800b32968"
server_key_version: "encryptionkey@0"
I20260806 00:58:15.236171 29370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance:
uuid: "ffdfba0e9ea44e528f8d6d55415afd40"
format_stamp: "Formatted at 2026-08-06 00:58:15 on dist-test-slave-dhph"
server_key: "a4e68a4260fed5094f408b5cc5c2d80e"
server_key_iv: "9b98c9af6e73865916d446d800b32968"
server_key_version: "encryptionkey@0"
I20260806 00:58:15.237610 29370 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 00:58:15.238734 29384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:15.239219 29370 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:15.239375 29370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "ffdfba0e9ea44e528f8d6d55415afd40"
format_stamp: "Formatted at 2026-08-06 00:58:15 on dist-test-slave-dhph"
server_key: "a4e68a4260fed5094f408b5cc5c2d80e"
server_key_iv: "9b98c9af6e73865916d446d800b32968"
server_key_version: "encryptionkey@0"
I20260806 00:58:15.239471 29370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:15.243324 29370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:15.243618 29370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:15.243765 29370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:15.244006 29370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:15.244467 29370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:15.244545 29370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:15.244587 29370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:15.244613 29370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:15.251263 29370 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:36843
I20260806 00:58:15.251411 29501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:36843 every 8 connection(s)
I20260806 00:58:15.251696 29370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 00:58:15.254837 29348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:15.252486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:56133 (local address 127.23.242.62:33071)
0806 00:58:15.252538 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:15.252540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:15.252754 (+ 214us) server_negotiation.cc:407] Connection header received
0806 00:58:15.253259 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:15.253260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:15.253266 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:15.253339 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:15.254521 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:15.254522 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:15.254523 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 00:58:15.254527 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:15.254542 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 00:58:15.254547 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:15.254673 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:15.254763 (+ 90us) server_negotiation.cc:299] Negotiation successful
0806 00:58:15.254776 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 00:58:15.255067 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:15.252410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:33071 (local address 127.23.242.1:56133)
0806 00:58:15.252631 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:15.252676 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:15.253150 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:15.253411 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:15.253415 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:15.253423 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:15.254353 (+ 930us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 00:58:15.254353 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:15.254382 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 00:58:15.254410 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:15.254684 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:15.254691 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:15.254694 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 00:58:15.254774 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 00:58:15.254831 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":92,"threads_started":1}
I20260806 00:58:15.255807 29502 heartbeater.cc:347] Connected to a master server at 127.23.242.62:33071
I20260806 00:58:15.255899 29502 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:15.256095 29502 heartbeater.cc:511] Master 127.23.242.62:33071 requested a full tablet report, sending...
I20260806 00:58:15.256392 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 29370
I20260806 00:58:15.256502 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance
I20260806 00:58:15.256688 24520 external_mini_cluster.cc:1428] Setting key 8ecca0684ad4ff23656aa176efe8f224
I20260806 00:58:15.256735 29308 ts_manager.cc:194] Registered new tserver with Master: ffdfba0e9ea44e528f8d6d55415afd40 (127.23.242.1:36843)
I20260806 00:58:15.257778 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:0
--local_ip_for_outbound_sockets=127.23.242.2
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:33071
--builtin_ntp_servers=127.23.242.20:41525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 00:58:15.349525 29505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:15.349828 29505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:15.349856 29505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:15.351096 29505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:15.351137 29505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:15.351164 29505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 00:58:15.352528 29505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:33071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:15.352797 29505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:15.353117 29505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:15.358726 29511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:15.358726 29510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:15.359140 29513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:15.359959 29505 server_base.cc:1034] running on GCE node
I20260806 00:58:15.360210 29505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:15.360559 29505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:15.361757 29505 hybrid_clock.cc:648] HybridClock initialized: now 1785977895361723 us; error 50 us; skew 500 ppm
I20260806 00:58:15.363394 29505 webserver.cc:460] Webserver started at http://127.23.242.2:46611/ using document root <none> and password file <none>
I20260806 00:58:15.363879 29505 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:15.363976 29505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:15.364176 29505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:15.365474 29505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-1/data/instance:
uuid: "f26c37f2773e4447bd5572f8f7badc7a"
format_stamp: "Formatted at 2026-08-06 00:58:15 on dist-test-slave-dhph"
server_key: "19af82ebaba53535b50585514b96d66a"
server_key_iv: "54011c9232e60ee3f6eec14f8580a7fb"
server_key_version: "encryptionkey@0"
I20260806 00:58:15.365885 29505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance:
uuid: "f26c37f2773e4447bd5572f8f7badc7a"
format_stamp: "Formatted at 2026-08-06 00:58:15 on dist-test-slave-dhph"
server_key: "19af82ebaba53535b50585514b96d66a"
server_key_iv: "54011c9232e60ee3f6eec14f8580a7fb"
server_key_version: "encryptionkey@0"
I20260806 00:58:15.367512 29505 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260806 00:58:15.368574 29519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:15.368975 29505 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:15.369211 29505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "f26c37f2773e4447bd5572f8f7badc7a"
format_stamp: "Formatted at 2026-08-06 00:58:15 on dist-test-slave-dhph"
server_key: "19af82ebaba53535b50585514b96d66a"
server_key_iv: "54011c9232e60ee3f6eec14f8580a7fb"
server_key_version: "encryptionkey@0"
I20260806 00:58:15.369320 29505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:15.373363 29505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:15.373692 29505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:15.373845 29505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:15.374163 29505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:15.374492 29505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:15.374526 29505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:15.374550 29505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:15.374563 29505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:15.381584 29505 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:44685
I20260806 00:58:15.381625 29635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:44685 every 8 connection(s)
I20260806 00:58:15.381976 29505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 00:58:15.384173 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 29505
I20260806 00:58:15.384326 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance
I20260806 00:58:15.384581 24520 external_mini_cluster.cc:1428] Setting key 3385a8c1818f1f1f9f2faf7b61bcfc40
I20260806 00:58:15.385984 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:0
--local_ip_for_outbound_sockets=127.23.242.3
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:33071
--builtin_ntp_servers=127.23.242.20:41525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 00:58:15.386301 29348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:15.382736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:43539 (local address 127.23.242.62:33071)
0806 00:58:15.382805 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:15.382808 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:15.383261 (+ 453us) server_negotiation.cc:407] Connection header received
0806 00:58:15.383985 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:15.383989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:15.384002 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:15.384127 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:15.385915 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:15.385917 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:15.385917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 00:58:15.385922 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:15.385943 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 00:58:15.385948 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:15.386013 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:15.386160 (+ 147us) server_negotiation.cc:299] Negotiation successful
0806 00:58:15.386178 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 00:58:15.386793 29638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:15.382768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:33071 (local address 127.23.242.2:43539)
0806 00:58:15.383008 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:15.383045 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:15.383713 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:15.384228 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:15.384232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:15.384242 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:15.385691 (+ 1449us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 00:58:15.385692 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:15.385731 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 00:58:15.385768 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:15.386033 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:15.386040 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:15.386044 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 00:58:15.386416 (+ 372us) client_negotiation.cc:240] Negotiation successful
0806 00:58:15.386488 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":101,"threads_started":1}
I20260806 00:58:15.387295 29636 heartbeater.cc:347] Connected to a master server at 127.23.242.62:33071
I20260806 00:58:15.387419 29636 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:15.387591 29636 heartbeater.cc:511] Master 127.23.242.62:33071 requested a full tablet report, sending...
I20260806 00:58:15.388007 29308 ts_manager.cc:194] Registered new tserver with Master: f26c37f2773e4447bd5572f8f7badc7a (127.23.242.2:44685)
W20260806 00:58:15.477075 29639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:15.477310 29639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:15.477336 29639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:15.478485 29639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:15.478524 29639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:15.478549 29639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 00:58:15.479871 29639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:33071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:15.480079 29639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:15.480340 29639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:15.485607 29645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:15.485725 29639 server_base.cc:1034] running on GCE node
W20260806 00:58:15.485651 29647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:15.485816 29644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:15.486058 29639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:15.486462 29639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:15.487673 29639 hybrid_clock.cc:648] HybridClock initialized: now 1785977895487665 us; error 41 us; skew 500 ppm
I20260806 00:58:15.489207 29639 webserver.cc:460] Webserver started at http://127.23.242.3:39349/ using document root <none> and password file <none>
I20260806 00:58:15.489490 29639 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:15.489566 29639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:15.489725 29639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:15.490940 29639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-2/data/instance:
uuid: "ac8d370016624b50a368e4d58a2301e7"
format_stamp: "Formatted at 2026-08-06 00:58:15 on dist-test-slave-dhph"
server_key: "edbab28ccd85659c420f14732b213c54"
server_key_iv: "06237bda5c1c61fb146c3d31fd575c5f"
server_key_version: "encryptionkey@0"
I20260806 00:58:15.491621 29639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance:
uuid: "ac8d370016624b50a368e4d58a2301e7"
format_stamp: "Formatted at 2026-08-06 00:58:15 on dist-test-slave-dhph"
server_key: "edbab28ccd85659c420f14732b213c54"
server_key_iv: "06237bda5c1c61fb146c3d31fd575c5f"
server_key_version: "encryptionkey@0"
I20260806 00:58:15.493203 29639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 00:58:15.494241 29653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:15.494458 29639 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:15.494573 29639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "ac8d370016624b50a368e4d58a2301e7"
format_stamp: "Formatted at 2026-08-06 00:58:15 on dist-test-slave-dhph"
server_key: "edbab28ccd85659c420f14732b213c54"
server_key_iv: "06237bda5c1c61fb146c3d31fd575c5f"
server_key_version: "encryptionkey@0"
I20260806 00:58:15.494740 29639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:15.498221 29639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:15.498489 29639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:15.498778 29639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:15.499212 29639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:15.499634 29639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:15.499783 29639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:15.499866 29639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:15.499897 29639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:15.508756 29639 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:36773
I20260806 00:58:15.508916 29769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:36773 every 8 connection(s)
I20260806 00:58:15.509234 29639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
I20260806 00:58:15.512712 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 29639
I20260806 00:58:15.512758 29348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:15.510132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:58207 (local address 127.23.242.62:33071)
0806 00:58:15.510202 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:15.510204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:15.510496 (+ 292us) server_negotiation.cc:407] Connection header received
0806 00:58:15.511071 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:15.511072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:15.511079 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:15.511149 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:15.512378 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:15.512380 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:15.512380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 00:58:15.512384 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:15.512398 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 00:58:15.512402 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:15.512517 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:15.512642 (+ 125us) server_negotiation.cc:299] Negotiation successful
0806 00:58:15.512661 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 00:58:15.512851 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance
I20260806 00:58:15.513010 29772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:15.510145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:33071 (local address 127.23.242.3:58207)
0806 00:58:15.510327 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:15.510361 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:15.510949 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:15.511202 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:15.511205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:15.511213 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:15.512224 (+ 1011us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 00:58:15.512224 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:15.512252 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 00:58:15.512279 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:15.512534 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:15.512541 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:15.512544 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 00:58:15.512629 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 00:58:15.512722 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
I20260806 00:58:15.513041 24520 external_mini_cluster.cc:1428] Setting key c79098a6e7af4fb668253e59010b167e
I20260806 00:58:15.513581 29770 heartbeater.cc:347] Connected to a master server at 127.23.242.62:33071
I20260806 00:58:15.513760 29770 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:15.513939 29770 heartbeater.cc:511] Master 127.23.242.62:33071 requested a full tablet report, sending...
I20260806 00:58:15.514374 29308 ts_manager.cc:194] Registered new tserver with Master: ac8d370016624b50a368e4d58a2301e7 (127.23.242.3:36773)
I20260806 00:58:15.526291 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 00:58:15.530437 29348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:15.528131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40992 (local address 127.23.242.62:33071)
0806 00:58:15.528176 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:15.528179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:15.528374 (+ 195us) server_negotiation.cc:407] Connection header received
0806 00:58:15.528407 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:15.528408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:15.528414 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:15.528490 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:15.529872 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:15.529873 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:15.529874 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 00:58:15.529878 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:15.529894 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 00:58:15.529900 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:15.529967 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:15.530232 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 00:58:15.530332 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 00:58:15.532759 29308 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 00:58:15.534488 29308 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 00:58:15.539902 29778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:15.537223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:36843 (local address 127.0.0.1:46754)
0806 00:58:15.538055 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:15.538074 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:15.538161 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:15.538636 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:15.538637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:15.538641 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:15.539286 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 00:58:15.539287 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:15.539308 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 00:58:15.539335 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:15.539675 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:15.539681 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:15.539684 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 00:58:15.539785 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 00:58:15.539809 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":3,"thread_start_us":102,"threads_started":1}
I20260806 00:58:15.539917 29782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:15.537495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46754 (local address 127.23.242.1:36843)
0806 00:58:15.537766 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:15.537771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:15.538127 (+ 356us) server_negotiation.cc:407] Connection header received
0806 00:58:15.538242 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:15.538246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:15.538256 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:15.538308 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:15.539428 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:15.539437 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:15.539437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 00:58:15.539442 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:15.539485 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 00:58:15.539510 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:15.539699 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:15.539800 (+ 101us) server_negotiation.cc:299] Negotiation successful
0806 00:58:15.539820 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":155,"threads_started":1}
I20260806 00:58:15.540292 29783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:15.537605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34772 (local address 127.23.242.2:44685)
0806 00:58:15.537979 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:15.537984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:15.537997 (+ 13us) server_negotiation.cc:407] Connection header received
0806 00:58:15.538028 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:15.538030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:15.538043 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:15.538125 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:15.539751 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:15.539760 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:15.539760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 00:58:15.539765 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:15.539804 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 00:58:15.539833 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:15.539900 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:15.540167 (+ 267us) server_negotiation.cc:299] Negotiation successful
0806 00:58:15.540186 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":245,"threads_started":1}
I20260806 00:58:15.540308 29781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:15.537396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46894 (local address 127.23.242.3:36773)
0806 00:58:15.537585 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:15.537590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:15.537898 (+ 308us) server_negotiation.cc:407] Connection header received
0806 00:58:15.537963 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:15.537967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:15.537986 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:15.539288 (+ 1302us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:15.539849 (+ 561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:15.539852 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:15.539852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 00:58:15.539857 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:15.539885 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 00:58:15.539899 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:15.540171 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:15.540208 (+ 37us) server_negotiation.cc:299] Negotiation successful
0806 00:58:15.540224 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":80,"threads_started":1}
I20260806 00:58:15.540079 29779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:15.537220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:44685 (local address 127.0.0.1:34772)
0806 00:58:15.537897 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:15.537908 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:15.537942 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:15.538636 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:15.538638 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:15.538642 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:15.539286 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 00:58:15.539287 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:15.539308 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 00:58:15.539335 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:15.539930 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:15.539931 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:15.539932 (+ 1us) client_negotiation.cc:769] Sending connection context
0806 00:58:15.539983 (+ 51us) client_negotiation.cc:240] Negotiation successful
0806 00:58:15.539998 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":1,"thread_start_us":63,"threads_started":1}
I20260806 00:58:15.540091 29780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:15.537220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:36773 (local address 127.0.0.1:46894)
0806 00:58:15.537662 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:15.537704 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:15.537784 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:15.538271 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:15.538289 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:15.538301 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:15.539363 (+ 1062us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 00:58:15.539363 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:15.539372 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 00:58:15.539386 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:15.539986 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:15.539986 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:15.539987 (+ 1us) client_negotiation.cc:769] Sending connection context
0806 00:58:15.540016 (+ 29us) client_negotiation.cc:240] Negotiation successful
0806 00:58:15.540026 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":66,"threads_started":1}
I20260806 00:58:15.542189 29433 tablet_service.cc:1467] Processing CreateTablet for tablet 188a7d34f112484392603023726d2cbe (DEFAULT_TABLE table=test-table [id=ae1b9742088c45b887c7dade2109677a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:15.542256 29568 tablet_service.cc:1467] Processing CreateTablet for tablet 188a7d34f112484392603023726d2cbe (DEFAULT_TABLE table=test-table [id=ae1b9742088c45b887c7dade2109677a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:15.542256 29703 tablet_service.cc:1467] Processing CreateTablet for tablet 188a7d34f112484392603023726d2cbe (DEFAULT_TABLE table=test-table [id=ae1b9742088c45b887c7dade2109677a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:15.542649 29433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188a7d34f112484392603023726d2cbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:15.542665 29703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188a7d34f112484392603023726d2cbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:15.542665 29568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 188a7d34f112484392603023726d2cbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:15.546950 29786 tablet_bootstrap.cc:492] T 188a7d34f112484392603023726d2cbe P ffdfba0e9ea44e528f8d6d55415afd40: Bootstrap starting.
I20260806 00:58:15.547588 29786 tablet_bootstrap.cc:654] T 188a7d34f112484392603023726d2cbe P ffdfba0e9ea44e528f8d6d55415afd40: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:15.547914 29786 log.cc:824] T 188a7d34f112484392603023726d2cbe P ffdfba0e9ea44e528f8d6d55415afd40: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:15.548243 29784 tablet_bootstrap.cc:492] T 188a7d34f112484392603023726d2cbe P f26c37f2773e4447bd5572f8f7badc7a: Bootstrap starting.
I20260806 00:58:15.548847 29786 tablet_bootstrap.cc:492] T 188a7d34f112484392603023726d2cbe P ffdfba0e9ea44e528f8d6d55415afd40: No bootstrap required, opened a new log
I20260806 00:58:15.548930 29786 ts_tablet_manager.cc:1397] T 188a7d34f112484392603023726d2cbe P ffdfba0e9ea44e528f8d6d55415afd40: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 00:58:15.549000 29784 tablet_bootstrap.cc:654] T 188a7d34f112484392603023726d2cbe P f26c37f2773e4447bd5572f8f7badc7a: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:15.549299 29784 log.cc:824] T 188a7d34f112484392603023726d2cbe P f26c37f2773e4447bd5572f8f7badc7a: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:15.549624 29785 tablet_bootstrap.cc:492] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7: Bootstrap starting.
I20260806 00:58:15.550127 29784 tablet_bootstrap.cc:492] T 188a7d34f112484392603023726d2cbe P f26c37f2773e4447bd5572f8f7badc7a: No bootstrap required, opened a new log
I20260806 00:58:15.550211 29784 ts_tablet_manager.cc:1397] T 188a7d34f112484392603023726d2cbe P f26c37f2773e4447bd5572f8f7badc7a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 00:58:15.550302 29785 tablet_bootstrap.cc:654] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:15.550632 29785 log.cc:824] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:15.551471 29785 tablet_bootstrap.cc:492] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7: No bootstrap required, opened a new log
I20260806 00:58:15.551545 29785 ts_tablet_manager.cc:1397] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260806 00:58:15.551856 29786 raft_consensus.cc:348] T 188a7d34f112484392603023726d2cbe P ffdfba0e9ea44e528f8d6d55415afd40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26c37f2773e4447bd5572f8f7badc7a" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 44685 } } peers { permanent_uuid: "ac8d370016624b50a368e4d58a2301e7" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36773 } } peers { permanent_uuid: "ffdfba0e9ea44e528f8d6d55415afd40" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36843 } }
I20260806 00:58:15.552029 29786 raft_consensus.cc:374] T 188a7d34f112484392603023726d2cbe P ffdfba0e9ea44e528f8d6d55415afd40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:15.552076 29786 raft_consensus.cc:729] T 188a7d34f112484392603023726d2cbe P ffdfba0e9ea44e528f8d6d55415afd40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffdfba0e9ea44e528f8d6d55415afd40, State: Initialized, Role: FOLLOWER
I20260806 00:58:15.552229 29786 consensus_queue.cc:260] T 188a7d34f112484392603023726d2cbe P ffdfba0e9ea44e528f8d6d55415afd40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26c37f2773e4447bd5572f8f7badc7a" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 44685 } } peers { permanent_uuid: "ac8d370016624b50a368e4d58a2301e7" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36773 } } peers { permanent_uuid: "ffdfba0e9ea44e528f8d6d55415afd40" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36843 } }
I20260806 00:58:15.552531 29786 ts_tablet_manager.cc:1428] T 188a7d34f112484392603023726d2cbe P ffdfba0e9ea44e528f8d6d55415afd40: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260806 00:58:15.552678 29502 heartbeater.cc:503] Master 127.23.242.62:33071 was elected leader, sending a full tablet report...
I20260806 00:58:15.552924 29784 raft_consensus.cc:348] T 188a7d34f112484392603023726d2cbe P f26c37f2773e4447bd5572f8f7badc7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26c37f2773e4447bd5572f8f7badc7a" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 44685 } } peers { permanent_uuid: "ac8d370016624b50a368e4d58a2301e7" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36773 } } peers { permanent_uuid: "ffdfba0e9ea44e528f8d6d55415afd40" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36843 } }
I20260806 00:58:15.553094 29784 raft_consensus.cc:374] T 188a7d34f112484392603023726d2cbe P f26c37f2773e4447bd5572f8f7badc7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:15.553135 29784 raft_consensus.cc:729] T 188a7d34f112484392603023726d2cbe P f26c37f2773e4447bd5572f8f7badc7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26c37f2773e4447bd5572f8f7badc7a, State: Initialized, Role: FOLLOWER
I20260806 00:58:15.553262 29784 consensus_queue.cc:260] T 188a7d34f112484392603023726d2cbe P f26c37f2773e4447bd5572f8f7badc7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26c37f2773e4447bd5572f8f7badc7a" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 44685 } } peers { permanent_uuid: "ac8d370016624b50a368e4d58a2301e7" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36773 } } peers { permanent_uuid: "ffdfba0e9ea44e528f8d6d55415afd40" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36843 } }
I20260806 00:58:15.553543 29784 ts_tablet_manager.cc:1428] T 188a7d34f112484392603023726d2cbe P f26c37f2773e4447bd5572f8f7badc7a: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260806 00:58:15.553574 29636 heartbeater.cc:503] Master 127.23.242.62:33071 was elected leader, sending a full tablet report...
I20260806 00:58:15.554533 29785 raft_consensus.cc:348] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26c37f2773e4447bd5572f8f7badc7a" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 44685 } } peers { permanent_uuid: "ac8d370016624b50a368e4d58a2301e7" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36773 } } peers { permanent_uuid: "ffdfba0e9ea44e528f8d6d55415afd40" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36843 } }
I20260806 00:58:15.554741 29785 raft_consensus.cc:374] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:15.554792 29785 raft_consensus.cc:729] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac8d370016624b50a368e4d58a2301e7, State: Initialized, Role: FOLLOWER
I20260806 00:58:15.554906 29785 consensus_queue.cc:260] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26c37f2773e4447bd5572f8f7badc7a" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 44685 } } peers { permanent_uuid: "ac8d370016624b50a368e4d58a2301e7" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36773 } } peers { permanent_uuid: "ffdfba0e9ea44e528f8d6d55415afd40" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36843 } }
I20260806 00:58:15.555172 29785 ts_tablet_manager.cc:1428] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260806 00:58:15.555226 29770 heartbeater.cc:503] Master 127.23.242.62:33071 was elected leader, sending a full tablet report...
I20260806 00:58:15.584419 29792 raft_consensus.cc:482] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 00:58:15.584648 29792 raft_consensus.cc:504] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26c37f2773e4447bd5572f8f7badc7a" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 44685 } } peers { permanent_uuid: "ac8d370016624b50a368e4d58a2301e7" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36773 } } peers { permanent_uuid: "ffdfba0e9ea44e528f8d6d55415afd40" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36843 } }
I20260806 00:58:15.585019 29792 leader_election.cc:288] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f26c37f2773e4447bd5572f8f7badc7a (127.23.242.2:44685), ffdfba0e9ea44e528f8d6d55415afd40 (127.23.242.1:36843)
I20260806 00:58:15.586776 29783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:15.585487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:51125 (local address 127.23.242.2:44685)
0806 00:58:15.585541 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:15.585543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:15.585746 (+ 203us) server_negotiation.cc:407] Connection header received
0806 00:58:15.585852 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:15.585853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:15.585857 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:15.585894 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:15.586541 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:15.586542 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:15.586542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 00:58:15.586547 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:15.586563 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 00:58:15.586568 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:15.586642 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:15.586700 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 00:58:15.586712 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 00:58:15.586776 29782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:15.585516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:46407 (local address 127.23.242.1:36843)
0806 00:58:15.585577 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:15.585579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:15.585587 (+ 8us) server_negotiation.cc:407] Connection header received
0806 00:58:15.585621 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:15.585622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:15.585626 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:15.585746 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:15.586458 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:15.586459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:15.586459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 00:58:15.586463 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:15.586478 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 00:58:15.586505 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:15.586569 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:15.586683 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 00:58:15.586699 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 00:58:15.586838 29772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:15.585338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:36843 (local address 127.23.242.3:46407)
0806 00:58:15.585431 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:15.585446 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:15.585505 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:15.585793 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:15.585794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:15.585798 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:15.586324 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 00:58:15.586324 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:15.586333 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 00:58:15.586377 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:15.586617 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:15.586618 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:15.586619 (+ 1us) client_negotiation.cc:769] Sending connection context
0806 00:58:15.586675 (+ 56us) client_negotiation.cc:240] Negotiation successful
0806 00:58:15.586691 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 00:58:15.586860 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:15.585338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:44685 (local address 127.23.242.3:51125)
0806 00:58:15.585630 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:15.585687 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:15.585778 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:15.585920 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:15.585921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:15.585924 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:15.586429 (+ 505us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 00:58:15.586430 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:15.586436 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 00:58:15.586447 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:15.586651 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:15.586651 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:15.586652 (+ 1us) client_negotiation.cc:769] Sending connection context
0806 00:58:15.586699 (+ 47us) client_negotiation.cc:240] Negotiation successful
0806 00:58:15.586709 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":117,"thread_start_us":116,"threads_started":1}
I20260806 00:58:15.587343 29454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188a7d34f112484392603023726d2cbe" candidate_uuid: "ac8d370016624b50a368e4d58a2301e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffdfba0e9ea44e528f8d6d55415afd40" is_pre_election: true
I20260806 00:58:15.587502 29454 raft_consensus.cc:2454] T 188a7d34f112484392603023726d2cbe P ffdfba0e9ea44e528f8d6d55415afd40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac8d370016624b50a368e4d58a2301e7 in term 0.
I20260806 00:58:15.587481 29589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188a7d34f112484392603023726d2cbe" candidate_uuid: "ac8d370016624b50a368e4d58a2301e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26c37f2773e4447bd5572f8f7badc7a" is_pre_election: true
I20260806 00:58:15.587622 29589 raft_consensus.cc:2454] T 188a7d34f112484392603023726d2cbe P f26c37f2773e4447bd5572f8f7badc7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac8d370016624b50a368e4d58a2301e7 in term 0.
I20260806 00:58:15.587759 29657 leader_election.cc:302] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac8d370016624b50a368e4d58a2301e7, ffdfba0e9ea44e528f8d6d55415afd40; no voters:
I20260806 00:58:15.587921 29792 raft_consensus.cc:2781] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 00:58:15.587981 29792 raft_consensus.cc:482] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 00:58:15.588027 29792 raft_consensus.cc:3037] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:15.588759 29792 raft_consensus.cc:504] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26c37f2773e4447bd5572f8f7badc7a" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 44685 } } peers { permanent_uuid: "ac8d370016624b50a368e4d58a2301e7" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36773 } } peers { permanent_uuid: "ffdfba0e9ea44e528f8d6d55415afd40" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36843 } }
I20260806 00:58:15.588904 29792 leader_election.cc:288] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7 [CANDIDATE]: Term 1 election: Requested vote from peers f26c37f2773e4447bd5572f8f7badc7a (127.23.242.2:44685), ffdfba0e9ea44e528f8d6d55415afd40 (127.23.242.1:36843)
I20260806 00:58:15.589044 29454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188a7d34f112484392603023726d2cbe" candidate_uuid: "ac8d370016624b50a368e4d58a2301e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffdfba0e9ea44e528f8d6d55415afd40"
I20260806 00:58:15.589069 29589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "188a7d34f112484392603023726d2cbe" candidate_uuid: "ac8d370016624b50a368e4d58a2301e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f26c37f2773e4447bd5572f8f7badc7a"
I20260806 00:58:15.589133 29454 raft_consensus.cc:3037] T 188a7d34f112484392603023726d2cbe P ffdfba0e9ea44e528f8d6d55415afd40 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:15.589138 29589 raft_consensus.cc:3037] T 188a7d34f112484392603023726d2cbe P f26c37f2773e4447bd5572f8f7badc7a [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:15.590229 29589 raft_consensus.cc:2454] T 188a7d34f112484392603023726d2cbe P f26c37f2773e4447bd5572f8f7badc7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac8d370016624b50a368e4d58a2301e7 in term 1.
I20260806 00:58:15.590234 29454 raft_consensus.cc:2454] T 188a7d34f112484392603023726d2cbe P ffdfba0e9ea44e528f8d6d55415afd40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac8d370016624b50a368e4d58a2301e7 in term 1.
I20260806 00:58:15.590464 29656 leader_election.cc:302] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac8d370016624b50a368e4d58a2301e7, f26c37f2773e4447bd5572f8f7badc7a; no voters:
I20260806 00:58:15.590562 29792 raft_consensus.cc:2781] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:15.590777 29792 raft_consensus.cc:686] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7 [term 1 LEADER]: Becoming Leader. State: Replica: ac8d370016624b50a368e4d58a2301e7, State: Running, Role: LEADER
I20260806 00:58:15.590898 29792 consensus_queue.cc:237] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26c37f2773e4447bd5572f8f7badc7a" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 44685 } } peers { permanent_uuid: "ac8d370016624b50a368e4d58a2301e7" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36773 } } peers { permanent_uuid: "ffdfba0e9ea44e528f8d6d55415afd40" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36843 } }
I20260806 00:58:15.591800 29308 catalog_manager.cc:5495] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac8d370016624b50a368e4d58a2301e7 (127.23.242.3). New cstate: current_term: 1 leader_uuid: "ac8d370016624b50a368e4d58a2301e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f26c37f2773e4447bd5572f8f7badc7a" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 44685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac8d370016624b50a368e4d58a2301e7" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ffdfba0e9ea44e528f8d6d55415afd40" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36843 } health_report { overall_health: UNKNOWN } } }
I20260806 00:58:15.603401 29781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:15.602252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46900 (local address 127.23.242.3:36773)
0806 00:58:15.602318 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:15.602320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:15.602329 (+ 9us) server_negotiation.cc:407] Connection header received
0806 00:58:15.602388 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:15.602389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:15.602401 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:15.602452 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:15.603145 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:15.603146 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:15.603147 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 00:58:15.603151 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:15.603167 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 00:58:15.603171 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:15.603226 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:15.603276 (+ 50us) server_negotiation.cc:299] Negotiation successful
0806 00:58:15.603287 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 00:58:15.605904 29454 raft_consensus.cc:1261] T 188a7d34f112484392603023726d2cbe P ffdfba0e9ea44e528f8d6d55415afd40 [term 1 FOLLOWER]: Refusing update from remote peer ac8d370016624b50a368e4d58a2301e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:15.605930 29589 raft_consensus.cc:1261] T 188a7d34f112484392603023726d2cbe P f26c37f2773e4447bd5572f8f7badc7a [term 1 FOLLOWER]: Refusing update from remote peer ac8d370016624b50a368e4d58a2301e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:15.606343 29794 consensus_queue.cc:1035] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffdfba0e9ea44e528f8d6d55415afd40" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 36843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:15.606477 29792 consensus_queue.cc:1035] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f26c37f2773e4447bd5572f8f7badc7a" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 44685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:15.609467 29797 mvcc.cc:205] Tried to move back new op lower bound from 7315365460398616576 to 7315365460341153792. Current Snapshot: MvccSnapshot[applied={T|T < 7315365460398616576}]
I20260806 00:58:15.610188 29799 mvcc.cc:205] Tried to move back new op lower bound from 7315365460398616576 to 7315365460341153792. Current Snapshot: MvccSnapshot[applied={T|T < 7315365460398616576}]
I20260806 00:58:15.610705 29798 mvcc.cc:205] Tried to move back new op lower bound from 7315365460398616576 to 7315365460341153792. Current Snapshot: MvccSnapshot[applied={T|T < 7315365460398616576}]
I20260806 00:58:15.613323 29308 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:58:15.613454 29308 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:58:15.614493 29308 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a3fb9e3147546f5a701259705f3c81d: Sending DeleteTablet for 3 replicas of tablet 188a7d34f112484392603023726d2cbe
I20260806 00:58:15.615092 29568 tablet_service.cc:1514] Processing DeleteTablet for tablet 188a7d34f112484392603023726d2cbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:15 UTC) from {username='slave'} at 127.0.0.1:34772
I20260806 00:58:15.615382 29703 tablet_service.cc:1514] Processing DeleteTablet for tablet 188a7d34f112484392603023726d2cbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:15 UTC) from {username='slave'} at 127.0.0.1:46894
I20260806 00:58:15.615491 29433 tablet_service.cc:1514] Processing DeleteTablet for tablet 188a7d34f112484392603023726d2cbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:15 UTC) from {username='slave'} at 127.0.0.1:46754
I20260806 00:58:15.616073 29809 tablet_replica.cc:330] T 188a7d34f112484392603023726d2cbe P ffdfba0e9ea44e528f8d6d55415afd40: stopping tablet replica
I20260806 00:58:15.616228 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 29370
I20260806 00:58:15.616387 29809 raft_consensus.cc:2229] T 188a7d34f112484392603023726d2cbe P ffdfba0e9ea44e528f8d6d55415afd40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:15.616931 29807 tablet_replica.cc:330] T 188a7d34f112484392603023726d2cbe P f26c37f2773e4447bd5572f8f7badc7a: stopping tablet replica
I20260806 00:58:15.617048 29807 raft_consensus.cc:2229] T 188a7d34f112484392603023726d2cbe P f26c37f2773e4447bd5572f8f7badc7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:15.617156 29807 raft_consensus.cc:2258] T 188a7d34f112484392603023726d2cbe P f26c37f2773e4447bd5572f8f7badc7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:15.617352 29808 tablet_replica.cc:330] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7: stopping tablet replica
I20260806 00:58:15.617460 29808 raft_consensus.cc:2229] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7 [term 1 LEADER]: Raft consensus shutting down.
I20260806 00:58:15.617590 29808 raft_consensus.cc:2258] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:15.618026 29808 ts_tablet_manager.cc:1905] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:15.618115 29807 ts_tablet_manager.cc:1905] T 188a7d34f112484392603023726d2cbe P f26c37f2773e4447bd5572f8f7badc7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:15.619976 29808 ts_tablet_manager.cc:1918] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 00:58:15.620083 29808 log.cc:1196] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-2/wal/wals/188a7d34f112484392603023726d2cbe
I20260806 00:58:15.620362 29807 ts_tablet_manager.cc:1918] T 188a7d34f112484392603023726d2cbe P f26c37f2773e4447bd5572f8f7badc7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 00:58:15.620453 29807 log.cc:1196] T 188a7d34f112484392603023726d2cbe P f26c37f2773e4447bd5572f8f7badc7a: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785977869987768-24520-0/minicluster-data/ts-1/wal/wals/188a7d34f112484392603023726d2cbe
I20260806 00:58:15.620474 29808 ts_tablet_manager.cc:1939] T 188a7d34f112484392603023726d2cbe P ac8d370016624b50a368e4d58a2301e7: Deleting consensus metadata
I20260806 00:58:15.620770 29807 ts_tablet_manager.cc:1939] T 188a7d34f112484392603023726d2cbe P f26c37f2773e4447bd5572f8f7badc7a: Deleting consensus metadata
I20260806 00:58:15.621078 29292 catalog_manager.cc:4841] TS ac8d370016624b50a368e4d58a2301e7 (127.23.242.3:36773): tablet 188a7d34f112484392603023726d2cbe (table test-table [id=ae1b9742088c45b887c7dade2109677a]) successfully deleted
I20260806 00:58:15.621516 29293 catalog_manager.cc:4841] TS f26c37f2773e4447bd5572f8f7badc7a (127.23.242.2:44685): tablet 188a7d34f112484392603023726d2cbe (table test-table [id=ae1b9742088c45b887c7dade2109677a]) successfully deleted
W20260806 00:58:15.625691 29294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.242.1:36843 (error 108)
I20260806 00:58:15.625845 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 29505
I20260806 00:58:15.631222 29780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:15.630999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:36843 (local address 127.0.0.1:46768)
0806 00:58:15.631089 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:15.631165 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.1:36843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 00:58:15.631467 29294 catalog_manager.cc:4584] TS ffdfba0e9ea44e528f8d6d55415afd40 (127.23.242.1:36843): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 188a7d34f112484392603023726d2cbe: Network error: Client connection negotiation failed: client connection to 127.23.242.1:36843: connect: Connection refused (error 111)
I20260806 00:58:15.631973 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 29639
I20260806 00:58:15.639806 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 29276
2026-08-06T00:58:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (764 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 00:58:15.662416 24520 mini_oidc.cc:152] Starting JWKS server
I20260806 00:58:15.664034 24520 webserver.cc:460] Webserver started at https://127.0.0.1:40623/ using document root <none> and password file <none>
I20260806 00:58:15.664124 24520 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 00:58:15.664273 24520 webserver.cc:460] Webserver started at http://127.0.0.1:44219/ using document root <none> and password file <none>
2026-08-06T00:58:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T00:58:15Z Disabled control of system clock
I20260806 00:58:15.673774 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:38049
--webserver_interface=127.23.242.62
--webserver_port=0
--builtin_ntp_servers=127.23.242.20:46615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:38049
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40623/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785977869987768-24520-0/testchainca.cert with env {}
W20260806 00:58:15.765450 29817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:15.765676 29817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:15.765709 29817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:15.766909 29817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 00:58:15.766956 29817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:15.766970 29817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:15.766983 29817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 00:58:15.766995 29817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 00:58:15.767009 29817 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 00:58:15.767021 29817 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40623/jwks.json
I20260806 00:58:15.770328 29817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:46615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:38049
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:38049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40623/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785977869987768-24520-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:15.770725 29817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:15.770946 29817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:15.777980 29824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:15.777963 29822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:15.778007 29817 server_base.cc:1034] running on GCE node
W20260806 00:58:15.777963 29826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:15.778544 29817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:15.778985 29817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:15.780191 29817 hybrid_clock.cc:648] HybridClock initialized: now 1785977895780159 us; error 45 us; skew 500 ppm
I20260806 00:58:15.782064 29817 webserver.cc:460] Webserver started at http://127.23.242.62:44611/ using document root <none> and password file <none>
I20260806 00:58:15.782388 29817 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:15.782474 29817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:15.782683 29817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:15.783946 29817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "c7191e3456a041f1b463280b1bd2d2d7"
format_stamp: "Formatted at 2026-08-06 00:58:15 on dist-test-slave-dhph"
server_key: "5543e08adf1bfd642f0001a99a65cc59"
server_key_iv: "468b78e98210060afbc387665b1f5d98"
server_key_version: "encryptionkey@0"
I20260806 00:58:15.784360 29817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "c7191e3456a041f1b463280b1bd2d2d7"
format_stamp: "Formatted at 2026-08-06 00:58:15 on dist-test-slave-dhph"
server_key: "5543e08adf1bfd642f0001a99a65cc59"
server_key_iv: "468b78e98210060afbc387665b1f5d98"
server_key_version: "encryptionkey@0"
I20260806 00:58:15.785955 29817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 00:58:15.787115 29832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:15.787474 29817 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:15.787640 29817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "c7191e3456a041f1b463280b1bd2d2d7"
format_stamp: "Formatted at 2026-08-06 00:58:15 on dist-test-slave-dhph"
server_key: "5543e08adf1bfd642f0001a99a65cc59"
server_key_iv: "468b78e98210060afbc387665b1f5d98"
server_key_version: "encryptionkey@0"
I20260806 00:58:15.787799 29817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:15.806780 29817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:15.807139 29817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:15.807535 29817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:15.813380 29817 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:38049
I20260806 00:58:15.813460 29885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:38049 every 8 connection(s)
I20260806 00:58:15.813841 29817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 00:58:15.814952 29886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:15.817449 29886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7191e3456a041f1b463280b1bd2d2d7: Bootstrap starting.
I20260806 00:58:15.818333 29886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7191e3456a041f1b463280b1bd2d2d7: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:15.818812 29886 log.cc:824] T 00000000000000000000000000000000 P c7191e3456a041f1b463280b1bd2d2d7: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:15.819794 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 29817
I20260806 00:58:15.819921 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 00:58:15.820068 29886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7191e3456a041f1b463280b1bd2d2d7: No bootstrap required, opened a new log
I20260806 00:58:15.820104 24520 external_mini_cluster.cc:1428] Setting key 7f69caa0f531d74e052a2b83b04fe673
I20260806 00:58:15.822647 29886 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7191e3456a041f1b463280b1bd2d2d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7191e3456a041f1b463280b1bd2d2d7" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38049 } }
I20260806 00:58:15.822863 29886 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7191e3456a041f1b463280b1bd2d2d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:15.822902 29886 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7191e3456a041f1b463280b1bd2d2d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7191e3456a041f1b463280b1bd2d2d7, State: Initialized, Role: FOLLOWER
I20260806 00:58:15.823029 29886 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7191e3456a041f1b463280b1bd2d2d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7191e3456a041f1b463280b1bd2d2d7" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38049 } }
I20260806 00:58:15.823122 29886 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7191e3456a041f1b463280b1bd2d2d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 00:58:15.823159 29886 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7191e3456a041f1b463280b1bd2d2d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 00:58:15.823201 29886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7191e3456a041f1b463280b1bd2d2d7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:15.824740 29886 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7191e3456a041f1b463280b1bd2d2d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7191e3456a041f1b463280b1bd2d2d7" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38049 } }
I20260806 00:58:15.824846 29886 leader_election.cc:302] T 00000000000000000000000000000000 P c7191e3456a041f1b463280b1bd2d2d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7191e3456a041f1b463280b1bd2d2d7; no voters:
I20260806 00:58:15.825387 29886 leader_election.cc:288] T 00000000000000000000000000000000 P c7191e3456a041f1b463280b1bd2d2d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 00:58:15.825423 29891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7191e3456a041f1b463280b1bd2d2d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:15.825546 29891 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7191e3456a041f1b463280b1bd2d2d7 [term 1 LEADER]: Becoming Leader. State: Replica: c7191e3456a041f1b463280b1bd2d2d7, State: Running, Role: LEADER
I20260806 00:58:15.825646 29891 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7191e3456a041f1b463280b1bd2d2d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7191e3456a041f1b463280b1bd2d2d7" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38049 } }
I20260806 00:58:15.825702 29886 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7191e3456a041f1b463280b1bd2d2d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 00:58:15.826002 29891 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7191e3456a041f1b463280b1bd2d2d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7191e3456a041f1b463280b1bd2d2d7. Latest consensus state: current_term: 1 leader_uuid: "c7191e3456a041f1b463280b1bd2d2d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7191e3456a041f1b463280b1bd2d2d7" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38049 } } }
I20260806 00:58:15.826016 29892 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7191e3456a041f1b463280b1bd2d2d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7191e3456a041f1b463280b1bd2d2d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7191e3456a041f1b463280b1bd2d2d7" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38049 } } }
I20260806 00:58:15.826098 29891 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7191e3456a041f1b463280b1bd2d2d7 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:15.826149 29892 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7191e3456a041f1b463280b1bd2d2d7 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:15.826982 29900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 00:58:15.827574 29900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 00:58:15.829138 29890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:15.820744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57370 (local address 127.23.242.62:38049)
0806 00:58:15.821003 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:15.821013 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:15.821047 (+ 34us) server_negotiation.cc:407] Connection header received
0806 00:58:15.821566 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:15.821575 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:15.821657 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:15.821828 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:15.822173 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:15.824567 (+ 2394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:15.826200 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:15.826887 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:15.827040 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:15.828220 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:15.828225 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:15.828225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 00:58:15.828468 (+ 243us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:15.828513 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 00:58:15.828529 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:15.828574 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:15.828789 (+ 215us) server_negotiation.cc:299] Negotiation successful
0806 00:58:15.828882 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":116,"threads_started":1}
I20260806 00:58:15.829465 29900 catalog_manager.cc:1347] Generated new cluster ID: d761565711844f76999188c5abe63eb8
I20260806 00:58:15.829547 29900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 00:58:15.830968 24520 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 00:58:15.847903 29900 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 00:58:15.849437 29900 catalog_manager.cc:1504] Loading token signing keys...
I20260806 00:58:15.861865 29900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7191e3456a041f1b463280b1bd2d2d7: Generated new TSK 0
I20260806 00:58:15.862103 29900 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 00:58:15.897768 29916 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 00:58:15.898507 29916 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260806 00:58:15.898782 29915 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:15.873912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38049 (local address 127.0.0.1:57386)
0806 00:58:15.874269 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:15.874280 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:15.874358 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:15.874800 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:15.874801 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:15.874804 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0806 00:58:15.874882 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:15.874885 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:15.876250 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:15.876251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:15.877062 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:15.877067 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:15.877641 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:15.877642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:15.877706 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:15.877733 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 00:58:15.898055 (+ 20322us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40623/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 00:58:15.898192 (+ 137us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.242.62:38049: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40623/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":118,"threads_started":1}
I20260806 00:58:15.898929 29890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:15.874036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57386 (local address 127.23.242.62:38049)
0806 00:58:15.874182 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:15.874185 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:15.874368 (+ 183us) server_negotiation.cc:407] Connection header received
0806 00:58:15.874455 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:15.874456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:15.874472 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:15.874605 (+ 133us) server_negotiation.cc:226] Negotiated authn=JWT
0806 00:58:15.874979 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:15.876090 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:15.877172 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:15.877542 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:15.877623 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:15.877874 (+ 251us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 00:58:15.897850 (+ 19976us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40623/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 00:58:15.898866 (+ 1016us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57386: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40623/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":31}
W20260806 00:58:15.899027 29890 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57386: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40623/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.23.242.62:38049: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40623/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 00:58:15.899447 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 29817
2026-08-06T00:58:15Z chronyd exiting
I20260806 00:58:15.938179 24520 test_util.cc:175] -----------------------------------------------
I20260806 00:58:15.938275 24520 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785977869987768-24520-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (277 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 00:58:15.939239 24520 mini_oidc.cc:152] Starting JWKS server
I20260806 00:58:15.940245 24520 webserver.cc:460] Webserver started at https://127.0.0.1:36371/ using document root <none> and password file <none>
I20260806 00:58:15.940322 24520 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 00:58:15.940481 24520 webserver.cc:460] Webserver started at http://127.0.0.1:44297/ using document root <none> and password file <none>
2026-08-06T00:58:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T00:58:15Z Disabled control of system clock
I20260806 00:58:15.951017 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:37225
--webserver_interface=127.23.242.62
--webserver_port=0
--builtin_ntp_servers=127.23.242.20:44295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:37225
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36371/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785977869987768-24520-0/testchainca.cert with env {}
W20260806 00:58:16.056983 29923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:16.057276 29923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:16.057307 29923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:16.058881 29923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 00:58:16.058938 29923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:16.058954 29923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:16.058969 29923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 00:58:16.058983 29923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 00:58:16.058998 29923 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 00:58:16.059011 29923 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36371/jwks.json
I20260806 00:58:16.060858 29923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:44295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:37225
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:37225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36371/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785977869987768-24520-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:16.061250 29923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:16.061519 29923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:16.067570 29928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:16.067582 29929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:16.067582 29931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:16.068044 29923 server_base.cc:1034] running on GCE node
I20260806 00:58:16.068375 29923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:16.068711 29923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:16.069903 29923 hybrid_clock.cc:648] HybridClock initialized: now 1785977896069885 us; error 30 us; skew 500 ppm
I20260806 00:58:16.071457 29923 webserver.cc:460] Webserver started at http://127.23.242.62:42941/ using document root <none> and password file <none>
I20260806 00:58:16.071815 29923 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:16.071902 29923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:16.072073 29923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:16.073354 29923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "252f9167442e4dee9b3006948734de80"
format_stamp: "Formatted at 2026-08-06 00:58:16 on dist-test-slave-dhph"
server_key: "aa3ecd921837662a06bf39c80644b120"
server_key_iv: "fbed3978a1c750b4264423f36bab7ae3"
server_key_version: "encryptionkey@0"
I20260806 00:58:16.073746 29923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "252f9167442e4dee9b3006948734de80"
format_stamp: "Formatted at 2026-08-06 00:58:16 on dist-test-slave-dhph"
server_key: "aa3ecd921837662a06bf39c80644b120"
server_key_iv: "fbed3978a1c750b4264423f36bab7ae3"
server_key_version: "encryptionkey@0"
I20260806 00:58:16.075328 29923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260806 00:58:16.076354 29937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:16.076753 29923 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:16.077039 29923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "252f9167442e4dee9b3006948734de80"
format_stamp: "Formatted at 2026-08-06 00:58:16 on dist-test-slave-dhph"
server_key: "aa3ecd921837662a06bf39c80644b120"
server_key_iv: "fbed3978a1c750b4264423f36bab7ae3"
server_key_version: "encryptionkey@0"
I20260806 00:58:16.077160 29923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:16.099767 29923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:16.100188 29923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:16.100370 29923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:16.107491 29923 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:37225
I20260806 00:58:16.107575 29991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:37225 every 8 connection(s)
I20260806 00:58:16.108018 29923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 00:58:16.109193 29992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:16.111897 29992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 252f9167442e4dee9b3006948734de80: Bootstrap starting.
I20260806 00:58:16.112576 29992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 252f9167442e4dee9b3006948734de80: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:16.112910 29992 log.cc:824] T 00000000000000000000000000000000 P 252f9167442e4dee9b3006948734de80: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:16.114123 29992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 252f9167442e4dee9b3006948734de80: No bootstrap required, opened a new log
I20260806 00:58:16.116145 29992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 252f9167442e4dee9b3006948734de80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252f9167442e4dee9b3006948734de80" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37225 } }
I20260806 00:58:16.116312 29992 raft_consensus.cc:374] T 00000000000000000000000000000000 P 252f9167442e4dee9b3006948734de80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:16.116338 29992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 252f9167442e4dee9b3006948734de80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252f9167442e4dee9b3006948734de80, State: Initialized, Role: FOLLOWER
I20260806 00:58:16.116516 29992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 252f9167442e4dee9b3006948734de80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252f9167442e4dee9b3006948734de80" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37225 } }
I20260806 00:58:16.116601 29992 raft_consensus.cc:388] T 00000000000000000000000000000000 P 252f9167442e4dee9b3006948734de80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 00:58:16.116638 29992 raft_consensus.cc:482] T 00000000000000000000000000000000 P 252f9167442e4dee9b3006948734de80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 00:58:16.116699 29992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 252f9167442e4dee9b3006948734de80 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:16.117388 29992 raft_consensus.cc:504] T 00000000000000000000000000000000 P 252f9167442e4dee9b3006948734de80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252f9167442e4dee9b3006948734de80" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37225 } }
I20260806 00:58:16.117484 29992 leader_election.cc:302] T 00000000000000000000000000000000 P 252f9167442e4dee9b3006948734de80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 252f9167442e4dee9b3006948734de80; no voters:
I20260806 00:58:16.117692 29992 leader_election.cc:288] T 00000000000000000000000000000000 P 252f9167442e4dee9b3006948734de80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 00:58:16.117949 29995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 252f9167442e4dee9b3006948734de80 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:16.118108 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 29923
I20260806 00:58:16.118155 29995 raft_consensus.cc:686] T 00000000000000000000000000000000 P 252f9167442e4dee9b3006948734de80 [term 1 LEADER]: Becoming Leader. State: Replica: 252f9167442e4dee9b3006948734de80, State: Running, Role: LEADER
I20260806 00:58:16.118238 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 00:58:16.118285 29992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 252f9167442e4dee9b3006948734de80 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 00:58:16.118429 24520 external_mini_cluster.cc:1428] Setting key 8014e7b8321d4c002c9513e22c6e9b0a
I20260806 00:58:16.118279 29995 consensus_queue.cc:237] T 00000000000000000000000000000000 P 252f9167442e4dee9b3006948734de80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252f9167442e4dee9b3006948734de80" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37225 } }
I20260806 00:58:16.119611 29996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 252f9167442e4dee9b3006948734de80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "252f9167442e4dee9b3006948734de80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252f9167442e4dee9b3006948734de80" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37225 } } }
I20260806 00:58:16.119715 29996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 252f9167442e4dee9b3006948734de80 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:16.120208 29997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 252f9167442e4dee9b3006948734de80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 252f9167442e4dee9b3006948734de80. Latest consensus state: current_term: 1 leader_uuid: "252f9167442e4dee9b3006948734de80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252f9167442e4dee9b3006948734de80" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37225 } } }
I20260806 00:58:16.120301 29997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 252f9167442e4dee9b3006948734de80 [sys.catalog]: This master's current role is: LEADER
W20260806 00:58:16.121277 30012 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 252f9167442e4dee9b3006948734de80: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 00:58:16.121340 30012 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 00:58:16.121387 30013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 00:58:16.121543 30013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 00:58:16.124102 30013 catalog_manager.cc:1347] Generated new cluster ID: b085c4f5432847669936cd7bf51ce56e
I20260806 00:58:16.124178 30013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 00:58:16.128049 30001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:16.118955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32882 (local address 127.23.242.62:37225)
0806 00:58:16.119347 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:16.119357 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:16.119391 (+ 34us) server_negotiation.cc:407] Connection header received
0806 00:58:16.119522 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:16.119530 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:16.119609 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:16.119851 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:16.120094 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:16.121487 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:16.122632 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:16.123355 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:16.123502 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:16.127082 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:16.127088 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:16.127088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 00:58:16.127381 (+ 293us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:16.127445 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 00:58:16.127465 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:16.127555 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:16.127694 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 00:58:16.127797 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":81,"threads_started":1}
I20260806 00:58:16.129341 30013 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 00:58:16.129438 24520 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 00:58:16.129927 30013 catalog_manager.cc:1504] Loading token signing keys...
I20260806 00:58:16.134642 30013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 252f9167442e4dee9b3006948734de80: Generated new TSK 0
I20260806 00:58:16.134891 30013 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 00:58:16.169006 30022 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 00:58:16.169114 30022 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 00:58:16.169430 30001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:16.142076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32890 (local address 127.23.242.62:37225)
0806 00:58:16.142208 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:16.142211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:16.142472 (+ 261us) server_negotiation.cc:407] Connection header received
0806 00:58:16.142523 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:16.142524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:16.142542 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:16.142706 (+ 164us) server_negotiation.cc:226] Negotiated authn=JWT
0806 00:58:16.142931 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:16.143439 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:16.144954 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:16.145239 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:16.145315 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:16.145611 (+ 296us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 00:58:16.169182 (+ 23571us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 00:58:16.169373 (+ 191us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32890: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":96}
W20260806 00:58:16.169507 30001 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32890: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260806 00:58:16.169589 30021 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:16.141962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:37225 (local address 127.0.0.1:32890)
0806 00:58:16.142321 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:16.142335 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:16.142422 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:16.142738 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:16.142739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:16.142741 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0806 00:58:16.142819 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:16.142822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:16.143619 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:16.143620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:16.144822 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:16.144826 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:16.145351 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:16.145352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:16.145429 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:16.145437 (+ 8us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 00:58:16.169337 (+ 23900us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 00:58:16.169481 (+ 144us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.242.62:37225: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36371/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":273,"threads_started":1}
I20260806 00:58:16.170009 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 29923
2026-08-06T00:58:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (270 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 00:58:16.209820 24520 mini_oidc.cc:152] Starting JWKS server
I20260806 00:58:16.210749 24520 webserver.cc:460] Webserver started at https://127.0.0.1:36265/ using document root <none> and password file <none>
I20260806 00:58:16.210810 24520 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 00:58:16.210983 24520 webserver.cc:460] Webserver started at http://127.0.0.1:33207/ using document root <none> and password file <none>
2026-08-06T00:58:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T00:58:16Z Disabled control of system clock
I20260806 00:58:16.221302 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:46349
--webserver_interface=127.23.242.62
--webserver_port=0
--builtin_ntp_servers=127.23.242.20:34711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:46349
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36265/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 00:58:16.315517 30028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:16.315753 30028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:16.315778 30028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:16.316958 30028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 00:58:16.317003 30028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:16.317016 30028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:16.317054 30028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 00:58:16.317070 30028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 00:58:16.317081 30028 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 00:58:16.317093 30028 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36265/jwks.json
I20260806 00:58:16.318926 30028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:34711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:46349
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:46349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36265/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:16.319200 30028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:16.319483 30028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:16.324617 30034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:16.324617 30036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:16.324739 30033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:16.325225 30028 server_base.cc:1034] running on GCE node
I20260806 00:58:16.325452 30028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:16.325843 30028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:16.327037 30028 hybrid_clock.cc:648] HybridClock initialized: now 1785977896326980 us; error 75 us; skew 500 ppm
I20260806 00:58:16.328823 30028 webserver.cc:460] Webserver started at http://127.23.242.62:33075/ using document root <none> and password file <none>
I20260806 00:58:16.329052 30028 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:16.329100 30028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:16.329197 30028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:16.330436 30028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "62f66c5dd0d245f2b3ca64528099accc"
format_stamp: "Formatted at 2026-08-06 00:58:16 on dist-test-slave-dhph"
server_key: "3c1846355b34483bb6c4073eebb173f9"
server_key_iv: "627ae1e37387bba0b86a10ca81b9654e"
server_key_version: "encryptionkey@0"
I20260806 00:58:16.330858 30028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "62f66c5dd0d245f2b3ca64528099accc"
format_stamp: "Formatted at 2026-08-06 00:58:16 on dist-test-slave-dhph"
server_key: "3c1846355b34483bb6c4073eebb173f9"
server_key_iv: "627ae1e37387bba0b86a10ca81b9654e"
server_key_version: "encryptionkey@0"
I20260806 00:58:16.332468 30028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:16.333436 30042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:16.333799 30028 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:16.333906 30028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "62f66c5dd0d245f2b3ca64528099accc"
format_stamp: "Formatted at 2026-08-06 00:58:16 on dist-test-slave-dhph"
server_key: "3c1846355b34483bb6c4073eebb173f9"
server_key_iv: "627ae1e37387bba0b86a10ca81b9654e"
server_key_version: "encryptionkey@0"
I20260806 00:58:16.333972 30028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:16.350188 30028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:16.350540 30028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:16.350826 30028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:16.355296 30028 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:46349
I20260806 00:58:16.355387 30098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:46349 every 8 connection(s)
I20260806 00:58:16.355736 30028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 00:58:16.356885 30099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:16.357360 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 30028
I20260806 00:58:16.357450 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 00:58:16.357690 24520 external_mini_cluster.cc:1428] Setting key 16326c1f711e62119cee2d14c19b59d3
I20260806 00:58:16.361152 30099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62f66c5dd0d245f2b3ca64528099accc: Bootstrap starting.
I20260806 00:58:16.361801 30099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62f66c5dd0d245f2b3ca64528099accc: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:16.362169 30099 log.cc:824] T 00000000000000000000000000000000 P 62f66c5dd0d245f2b3ca64528099accc: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:16.363217 30099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62f66c5dd0d245f2b3ca64528099accc: No bootstrap required, opened a new log
I20260806 00:58:16.365016 30103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:16.358323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47170 (local address 127.23.242.62:46349)
0806 00:58:16.358550 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:16.358559 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:16.358621 (+ 62us) server_negotiation.cc:407] Connection header received
0806 00:58:16.359127 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:16.359144 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:16.359236 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:16.359393 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:16.359688 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:16.361055 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:16.362417 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:16.362938 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:16.363076 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:16.364081 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:16.364086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:16.364087 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 00:58:16.364300 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:16.364335 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 00:58:16.364347 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:16.364434 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:16.364552 (+ 118us) server_negotiation.cc:299] Negotiation successful
0806 00:58:16.364644 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":85,"threads_started":1}
I20260806 00:58:16.365837 30099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62f66c5dd0d245f2b3ca64528099accc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f66c5dd0d245f2b3ca64528099accc" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 46349 } }
I20260806 00:58:16.366029 30099 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62f66c5dd0d245f2b3ca64528099accc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:16.366070 30099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62f66c5dd0d245f2b3ca64528099accc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f66c5dd0d245f2b3ca64528099accc, State: Initialized, Role: FOLLOWER
I20260806 00:58:16.366187 30099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62f66c5dd0d245f2b3ca64528099accc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f66c5dd0d245f2b3ca64528099accc" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 46349 } }
I20260806 00:58:16.366283 30099 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62f66c5dd0d245f2b3ca64528099accc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 00:58:16.366324 30099 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62f66c5dd0d245f2b3ca64528099accc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 00:58:16.366382 30099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62f66c5dd0d245f2b3ca64528099accc [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:16.366393 24520 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 00:58:16.367201 30099 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62f66c5dd0d245f2b3ca64528099accc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f66c5dd0d245f2b3ca64528099accc" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 46349 } }
I20260806 00:58:16.367352 30099 leader_election.cc:302] T 00000000000000000000000000000000 P 62f66c5dd0d245f2b3ca64528099accc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62f66c5dd0d245f2b3ca64528099accc; no voters:
I20260806 00:58:16.367548 30099 leader_election.cc:288] T 00000000000000000000000000000000 P 62f66c5dd0d245f2b3ca64528099accc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 00:58:16.367748 30105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62f66c5dd0d245f2b3ca64528099accc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:16.367837 30099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62f66c5dd0d245f2b3ca64528099accc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 00:58:16.368083 30105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62f66c5dd0d245f2b3ca64528099accc [term 1 LEADER]: Becoming Leader. State: Replica: 62f66c5dd0d245f2b3ca64528099accc, State: Running, Role: LEADER
I20260806 00:58:16.368180 30105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62f66c5dd0d245f2b3ca64528099accc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f66c5dd0d245f2b3ca64528099accc" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 46349 } }
I20260806 00:58:16.368563 30106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62f66c5dd0d245f2b3ca64528099accc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62f66c5dd0d245f2b3ca64528099accc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f66c5dd0d245f2b3ca64528099accc" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 46349 } } }
I20260806 00:58:16.368652 30106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62f66c5dd0d245f2b3ca64528099accc [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:16.368551 30107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62f66c5dd0d245f2b3ca64528099accc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62f66c5dd0d245f2b3ca64528099accc. Latest consensus state: current_term: 1 leader_uuid: "62f66c5dd0d245f2b3ca64528099accc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f66c5dd0d245f2b3ca64528099accc" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 46349 } } }
I20260806 00:58:16.368803 30107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62f66c5dd0d245f2b3ca64528099accc [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:16.368921 30114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 00:58:16.369539 30114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 00:58:16.371196 30114 catalog_manager.cc:1347] Generated new cluster ID: caff23a196fe47d68508c3b224a62eff
I20260806 00:58:16.371250 30114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 00:58:16.385955 30114 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 00:58:16.386803 30114 catalog_manager.cc:1504] Loading token signing keys...
I20260806 00:58:16.398433 30114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62f66c5dd0d245f2b3ca64528099accc: Generated new TSK 0
I20260806 00:58:16.398725 30114 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 00:58:16.410329 30103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:16.405895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47176 (local address 127.23.242.62:46349)
0806 00:58:16.405958 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:16.405960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:16.406064 (+ 104us) server_negotiation.cc:407] Connection header received
0806 00:58:16.406154 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:16.406155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:16.406171 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:16.406277 (+ 106us) server_negotiation.cc:226] Negotiated authn=JWT
0806 00:58:16.406448 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:16.407302 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:16.408388 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:16.408722 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:16.408806 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:16.409009 (+ 203us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 00:58:16.410079 (+ 1070us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36265/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 00:58:16.410238 (+ 159us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47176: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36265/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":36}
W20260806 00:58:16.410429 30129 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:16.405795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:46349 (local address 127.0.0.1:47176)
0806 00:58:16.405986 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:16.405997 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:16.406062 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:16.406306 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:16.406307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:16.406309 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0806 00:58:16.406363 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:16.406365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:16.407433 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:16.407434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:16.408281 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:16.408284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:16.408836 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:16.408837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:16.408886 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:16.408892 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 00:58:16.410203 (+ 1311us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36265/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 00:58:16.410315 (+ 112us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.242.62:46349: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36265/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":122,"threads_started":1}
W20260806 00:58:16.410791 30103 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47176: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36265/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260806 00:58:16.410887 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 30028
E20260806 00:58:16.422497 30130 webserver.cc:533] Webserver: sslize failed:
E20260806 00:58:16.422626 30130 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-06T00:58:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (241 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T00:58:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T00:58:16Z Disabled control of system clock
I20260806 00:58:16.459372 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:37871
--webserver_interface=127.23.242.62
--webserver_port=0
--builtin_ntp_servers=127.23.242.20:35093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:37871
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785977869987768-24520-0/testchainca.cert with env {}
W20260806 00:58:16.546491 30134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:16.546736 30134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:16.546761 30134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:16.547869 30134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 00:58:16.547905 30134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:16.547919 30134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:16.547930 30134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 00:58:16.547942 30134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 00:58:16.547955 30134 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 00:58:16.547966 30134 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 00:58:16.549440 30134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:35093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:37871
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:37871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785977869987768-24520-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:16.549654 30134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:16.549832 30134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:16.555004 30142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:16.555001 30140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:16.555094 30139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:16.555152 30134 server_base.cc:1034] running on GCE node
I20260806 00:58:16.555459 30134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:16.555765 30134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:16.556947 30134 hybrid_clock.cc:648] HybridClock initialized: now 1785977896556916 us; error 53 us; skew 500 ppm
I20260806 00:58:16.558310 30134 webserver.cc:460] Webserver started at http://127.23.242.62:36075/ using document root <none> and password file <none>
I20260806 00:58:16.558550 30134 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:16.558646 30134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:16.558804 30134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:16.559892 30134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "33236d39703f4f07a9fe5f6167550000"
format_stamp: "Formatted at 2026-08-06 00:58:16 on dist-test-slave-dhph"
server_key: "96de787304ed48d98ee610612a7e494d"
server_key_iv: "c1174744842a4e84d16ed8196174cf32"
server_key_version: "encryptionkey@0"
I20260806 00:58:16.560287 30134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "33236d39703f4f07a9fe5f6167550000"
format_stamp: "Formatted at 2026-08-06 00:58:16 on dist-test-slave-dhph"
server_key: "96de787304ed48d98ee610612a7e494d"
server_key_iv: "c1174744842a4e84d16ed8196174cf32"
server_key_version: "encryptionkey@0"
I20260806 00:58:16.561802 30134 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 00:58:16.562734 30148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:16.562937 30134 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:16.563054 30134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "33236d39703f4f07a9fe5f6167550000"
format_stamp: "Formatted at 2026-08-06 00:58:16 on dist-test-slave-dhph"
server_key: "96de787304ed48d98ee610612a7e494d"
server_key_iv: "c1174744842a4e84d16ed8196174cf32"
server_key_version: "encryptionkey@0"
I20260806 00:58:16.563176 30134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:16.574630 30134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:16.574986 30134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:16.575155 30134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:16.580049 30134 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:37871
I20260806 00:58:16.580094 30202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:37871 every 8 connection(s)
I20260806 00:58:16.580454 30134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 00:58:16.581533 30203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:16.584280 30203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33236d39703f4f07a9fe5f6167550000: Bootstrap starting.
I20260806 00:58:16.584311 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 30134
I20260806 00:58:16.584452 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 00:58:16.584666 24520 external_mini_cluster.cc:1428] Setting key bcf452592ec762f3a4cc3a4b00546367
I20260806 00:58:16.585083 30203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33236d39703f4f07a9fe5f6167550000: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:16.585486 30203 log.cc:824] T 00000000000000000000000000000000 P 33236d39703f4f07a9fe5f6167550000: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:16.586310 30203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33236d39703f4f07a9fe5f6167550000: No bootstrap required, opened a new log
I20260806 00:58:16.588361 30203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33236d39703f4f07a9fe5f6167550000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33236d39703f4f07a9fe5f6167550000" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37871 } }
I20260806 00:58:16.588546 30203 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33236d39703f4f07a9fe5f6167550000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:16.588603 30203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33236d39703f4f07a9fe5f6167550000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33236d39703f4f07a9fe5f6167550000, State: Initialized, Role: FOLLOWER
I20260806 00:58:16.588771 30203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33236d39703f4f07a9fe5f6167550000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33236d39703f4f07a9fe5f6167550000" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37871 } }
I20260806 00:58:16.588888 30203 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33236d39703f4f07a9fe5f6167550000 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 00:58:16.588939 30203 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33236d39703f4f07a9fe5f6167550000 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 00:58:16.588995 30203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33236d39703f4f07a9fe5f6167550000 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:16.589627 30203 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33236d39703f4f07a9fe5f6167550000 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33236d39703f4f07a9fe5f6167550000" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37871 } }
I20260806 00:58:16.589757 30203 leader_election.cc:302] T 00000000000000000000000000000000 P 33236d39703f4f07a9fe5f6167550000 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33236d39703f4f07a9fe5f6167550000; no voters:
I20260806 00:58:16.589969 30203 leader_election.cc:288] T 00000000000000000000000000000000 P 33236d39703f4f07a9fe5f6167550000 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 00:58:16.590005 30208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33236d39703f4f07a9fe5f6167550000 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:16.590193 30208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33236d39703f4f07a9fe5f6167550000 [term 1 LEADER]: Becoming Leader. State: Replica: 33236d39703f4f07a9fe5f6167550000, State: Running, Role: LEADER
I20260806 00:58:16.590305 30203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33236d39703f4f07a9fe5f6167550000 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 00:58:16.590291 30208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33236d39703f4f07a9fe5f6167550000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33236d39703f4f07a9fe5f6167550000" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37871 } }
I20260806 00:58:16.590688 30209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33236d39703f4f07a9fe5f6167550000 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33236d39703f4f07a9fe5f6167550000. Latest consensus state: current_term: 1 leader_uuid: "33236d39703f4f07a9fe5f6167550000" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33236d39703f4f07a9fe5f6167550000" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37871 } } }
I20260806 00:58:16.590806 30209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33236d39703f4f07a9fe5f6167550000 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:16.591034 30210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33236d39703f4f07a9fe5f6167550000 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33236d39703f4f07a9fe5f6167550000" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33236d39703f4f07a9fe5f6167550000" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37871 } } }
I20260806 00:58:16.591109 30210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33236d39703f4f07a9fe5f6167550000 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:16.591738 30206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:16.585461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52738 (local address 127.23.242.62:37871)
0806 00:58:16.585696 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:16.585710 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:16.585768 (+ 58us) server_negotiation.cc:407] Connection header received
0806 00:58:16.586329 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:16.586336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:16.586396 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:16.586612 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:16.586939 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:16.588010 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:16.589078 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:16.589549 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:16.589733 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:16.590606 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:16.590611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:16.590611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 00:58:16.590845 (+ 234us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:16.590895 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 00:58:16.590913 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:16.590997 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:16.591396 (+ 399us) server_negotiation.cc:299] Negotiation successful
0806 00:58:16.591484 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":93,"threads_started":1}
W20260806 00:58:16.592564 30224 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 33236d39703f4f07a9fe5f6167550000: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 00:58:16.592639 30224 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 00:58:16.592712 30213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 00:58:16.592780 24520 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 00:58:16.593734 30213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 00:58:16.599089 30213 catalog_manager.cc:1347] Generated new cluster ID: 97d1802823694d29a3c967af320ff83f
I20260806 00:58:16.599159 30213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 00:58:16.608191 30213 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 00:58:16.608764 30213 catalog_manager.cc:1504] Loading token signing keys...
I20260806 00:58:16.616029 30213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33236d39703f4f07a9fe5f6167550000: Generated new TSK 0
I20260806 00:58:16.616286 30213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 00:58:17.474038 30206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:16.621319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52750 (local address 127.23.242.62:37871)
0806 00:58:16.621369 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:16.621371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:16.621476 (+ 105us) server_negotiation.cc:407] Connection header received
0806 00:58:16.621539 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:16.621540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:16.621562 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:16.621669 (+ 107us) server_negotiation.cc:226] Negotiated authn=JWT
0806 00:58:16.621956 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:16.622485 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:16.623720 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:16.623937 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:16.624015 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:16.624203 (+ 188us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 00:58:17.473802 (+849599us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 00:58:17.473975 (+ 173us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52750: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":28}
W20260806 00:58:17.474105 30206 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52750: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260806 00:58:17.474299 30232 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:16.621192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:37871 (local address 127.0.0.1:52750)
0806 00:58:16.621356 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:16.621369 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:16.621441 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:16.621692 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:16.621693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:16.621694 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0806 00:58:16.621757 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:16.621759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:16.622599 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:16.622601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:16.623601 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:16.623604 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:16.624032 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:16.624033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:16.624096 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:16.624103 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 00:58:17.473997 (+849894us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 00:58:17.474171 (+ 174us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.242.62:37871: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":108,"threads_started":1}
I20260806 00:58:17.474668 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 30134
2026-08-06T00:58:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1044 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (108 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (100 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 00:58:17 dist-test-slave-dhph krb5kdc[30244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 00:58:17 dist-test-slave-dhph krb5kdc[30244](info): set up 2 sockets
Aug 06 00:58:17 dist-test-slave-dhph krb5kdc[30244](info): commencing operation
krb5kdc: starting...
W20260806 00:58:18.809131 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 00:58:18 dist-test-slave-dhph krb5kdc[30244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T00:58:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T00:58:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 00:58:18.960546 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:42293
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:45675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:42293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:19.051201 30258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:19.051434 30258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:19.051460 30258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:19.052613 30258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 00:58:19.052655 30258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:19.052670 30258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:19.052683 30258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 00:58:19.052695 30258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 00:58:19.054171 30258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:45675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:42293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:42293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:19.054381 30258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:19.054563 30258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:19.060009 30263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:19.060012 30264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:19.060065 30266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:19.060213 30258 server_base.cc:1034] running on GCE node
I20260806 00:58:19.060734 30258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:19.060997 30258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:19.062201 30258 hybrid_clock.cc:648] HybridClock initialized: now 1785977899062160 us; error 74 us; skew 500 ppm
Aug 06 00:58:19 dist-test-slave-dhph krb5kdc[30244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977899, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:19.070007 30258 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 00:58:19.070703 30258 webserver.cc:460] Webserver started at http://127.23.242.62:37835/ using document root <none> and password file <none>
I20260806 00:58:19.070971 30258 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:19.071058 30258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:19.071224 30258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:19.072420 30258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "51e84e54872d41aebb2e9630f82e5556"
format_stamp: "Formatted at 2026-08-06 00:58:19 on dist-test-slave-dhph"
server_key: "8c0537f34e0f60abdd6b4c0c4159756f"
server_key_iv: "f61e0d3f9d6fa1b3ee1eb4199ec5cbce"
server_key_version: "encryptionkey@0"
I20260806 00:58:19.072826 30258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "51e84e54872d41aebb2e9630f82e5556"
format_stamp: "Formatted at 2026-08-06 00:58:19 on dist-test-slave-dhph"
server_key: "8c0537f34e0f60abdd6b4c0c4159756f"
server_key_iv: "f61e0d3f9d6fa1b3ee1eb4199ec5cbce"
server_key_version: "encryptionkey@0"
I20260806 00:58:19.074404 30258 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 00:58:19.075482 30273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:19.075757 30258 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:19.075879 30258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "51e84e54872d41aebb2e9630f82e5556"
format_stamp: "Formatted at 2026-08-06 00:58:19 on dist-test-slave-dhph"
server_key: "8c0537f34e0f60abdd6b4c0c4159756f"
server_key_iv: "f61e0d3f9d6fa1b3ee1eb4199ec5cbce"
server_key_version: "encryptionkey@0"
I20260806 00:58:19.075960 30258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:19.101848 30258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:19.104413 30258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:19.104597 30258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:19.109191 30258 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:42293
I20260806 00:58:19.109244 30326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:42293 every 8 connection(s)
I20260806 00:58:19.109640 30258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 00:58:19.110778 30327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:19.113355 30327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51e84e54872d41aebb2e9630f82e5556: Bootstrap starting.
I20260806 00:58:19.114002 30327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51e84e54872d41aebb2e9630f82e5556: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:19.114346 30327 log.cc:824] T 00000000000000000000000000000000 P 51e84e54872d41aebb2e9630f82e5556: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:19.115185 30327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51e84e54872d41aebb2e9630f82e5556: No bootstrap required, opened a new log
I20260806 00:58:19.116914 30327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51e84e54872d41aebb2e9630f82e5556 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e84e54872d41aebb2e9630f82e5556" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 42293 } }
I20260806 00:58:19.117067 30327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51e84e54872d41aebb2e9630f82e5556 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:19.117092 30327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51e84e54872d41aebb2e9630f82e5556 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51e84e54872d41aebb2e9630f82e5556, State: Initialized, Role: FOLLOWER
I20260806 00:58:19.117234 30327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51e84e54872d41aebb2e9630f82e5556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e84e54872d41aebb2e9630f82e5556" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 42293 } }
I20260806 00:58:19.117313 30327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51e84e54872d41aebb2e9630f82e5556 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 00:58:19.117338 30327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51e84e54872d41aebb2e9630f82e5556 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 00:58:19.117401 30327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51e84e54872d41aebb2e9630f82e5556 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:19.118039 30327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51e84e54872d41aebb2e9630f82e5556 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e84e54872d41aebb2e9630f82e5556" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 42293 } }
I20260806 00:58:19.118032 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 30258
I20260806 00:58:19.118136 30327 leader_election.cc:302] T 00000000000000000000000000000000 P 51e84e54872d41aebb2e9630f82e5556 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51e84e54872d41aebb2e9630f82e5556; no voters:
I20260806 00:58:19.118209 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 00:58:19.118333 30327 leader_election.cc:288] T 00000000000000000000000000000000 P 51e84e54872d41aebb2e9630f82e5556 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 00:58:19.118367 30330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51e84e54872d41aebb2e9630f82e5556 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:19.118690 30327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51e84e54872d41aebb2e9630f82e5556 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 00:58:19.118760 24520 external_mini_cluster.cc:1428] Setting key a62f1dd964254a81f74166266b735f45
I20260806 00:58:19.118733 30330 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51e84e54872d41aebb2e9630f82e5556 [term 1 LEADER]: Becoming Leader. State: Replica: 51e84e54872d41aebb2e9630f82e5556, State: Running, Role: LEADER
I20260806 00:58:19.119037 30330 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51e84e54872d41aebb2e9630f82e5556 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e84e54872d41aebb2e9630f82e5556" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 42293 } }
I20260806 00:58:19.119457 30331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51e84e54872d41aebb2e9630f82e5556 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51e84e54872d41aebb2e9630f82e5556" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e84e54872d41aebb2e9630f82e5556" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 42293 } } }
I20260806 00:58:19.119553 30331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51e84e54872d41aebb2e9630f82e5556 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:19.120002 30332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51e84e54872d41aebb2e9630f82e5556 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51e84e54872d41aebb2e9630f82e5556. Latest consensus state: current_term: 1 leader_uuid: "51e84e54872d41aebb2e9630f82e5556" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51e84e54872d41aebb2e9630f82e5556" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 42293 } } }
I20260806 00:58:19.120157 30332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51e84e54872d41aebb2e9630f82e5556 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:19.120265 30342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 00:58:19.120858 30342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 00:58:19.122614 30342 catalog_manager.cc:1347] Generated new cluster ID: f4885df0aba148b780812f17e073484c
I20260806 00:58:19.122687 30342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 00:58:19 dist-test-slave-dhph krb5kdc[30244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:19.131681 30341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:19.119501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45226 (local address 127.23.242.62:42293)
0806 00:58:19.119713 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:19.119724 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:19.120387 (+ 663us) server_negotiation.cc:407] Connection header received
0806 00:58:19.120621 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:19.120629 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:19.120698 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:19.120859 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:19.121523 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:19.122858 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:19.123643 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:19.124106 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:19.124230 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:19.127104 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:19.127113 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:19.127113 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:19.127131 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:19.129774 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:19.130225 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:19.130234 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:19.130235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:19.130332 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:19.130613 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:19.130614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:19.130615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:19.130935 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:19.131135 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:19.131246 (+ 111us) server_negotiation.cc:299] Negotiation successful
0806 00:58:19.131427 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":92,"threads_started":1}
I20260806 00:58:19.151494 30342 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 00:58:19.152387 30342 catalog_manager.cc:1504] Loading token signing keys...
I20260806 00:58:19.162690 30342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51e84e54872d41aebb2e9630f82e5556: Generated new TSK 0
I20260806 00:58:19.162956 30342 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.242.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
I20260806 00:58:19.211171 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:0
--local_ip_for_outbound_sockets=127.23.242.1
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:42293
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:45675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:19.301304 30354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:19.301577 30354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:19.301622 30354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:19.302942 30354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:19.303010 30354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:19.303066 30354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 00:58:19.304837 30354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:45675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:42293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:19.305138 30354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:19.305398 30354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:19.311316 30360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:19.311419 30359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:19.311316 30362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:19.311448 30354 server_base.cc:1034] running on GCE node
I20260806 00:58:19.311705 30354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:19.312027 30354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:19.313397 30354 hybrid_clock.cc:648] HybridClock initialized: now 1785977899313373 us; error 37 us; skew 500 ppm
Aug 06 00:58:19 dist-test-slave-dhph krb5kdc[30244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977899, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:19.316460 30354 init.cc:377] Logged in from keytab as kudu/127.23.242.1@KRBTEST.COM (short username kudu)
I20260806 00:58:19.317011 30354 webserver.cc:460] Webserver started at http://127.23.242.1:45061/ using document root <none> and password file <none>
I20260806 00:58:19.317261 30354 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:19.317333 30354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:19.317492 30354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:19.318828 30354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-0/data/instance:
uuid: "628cdfe3d17246d1921f5720c9b905d2"
format_stamp: "Formatted at 2026-08-06 00:58:19 on dist-test-slave-dhph"
server_key: "07ca74d385e131c734d875b06d6a4c12"
server_key_iv: "2c7c267769fcd9fe04f92defd2043743"
server_key_version: "encryptionkey@0"
I20260806 00:58:19.319245 30354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance:
uuid: "628cdfe3d17246d1921f5720c9b905d2"
format_stamp: "Formatted at 2026-08-06 00:58:19 on dist-test-slave-dhph"
server_key: "07ca74d385e131c734d875b06d6a4c12"
server_key_iv: "2c7c267769fcd9fe04f92defd2043743"
server_key_version: "encryptionkey@0"
I20260806 00:58:19.321099 30354 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 00:58:19.322199 30369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:19.322779 30354 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:19.322894 30354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "628cdfe3d17246d1921f5720c9b905d2"
format_stamp: "Formatted at 2026-08-06 00:58:19 on dist-test-slave-dhph"
server_key: "07ca74d385e131c734d875b06d6a4c12"
server_key_iv: "2c7c267769fcd9fe04f92defd2043743"
server_key_version: "encryptionkey@0"
I20260806 00:58:19.322984 30354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:19.354298 30354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:19.357110 30354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:19.357331 30354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:19.357626 30354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:19.357992 30354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:19.358054 30354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:19.358100 30354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:19.358134 30354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:19.366107 30354 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:44785
I20260806 00:58:19.366154 30484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:44785 every 8 connection(s)
I20260806 00:58:19.366535 30354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 00:58:19.368659 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 30354
I20260806 00:58:19.368853 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance
I20260806 00:58:19.369100 24520 external_mini_cluster.cc:1428] Setting key 2de05ef9afcb1bed1ef25f9a47406638
Aug 06 00:58:19 dist-test-slave-dhph krb5kdc[30244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977899, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.242.2@KRBTEST.COM; defaulting to no policy
I20260806 00:58:19.380542 30341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:19.367748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:46813 (local address 127.23.242.62:42293)
0806 00:58:19.367826 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:19.367828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:19.368011 (+ 183us) server_negotiation.cc:407] Connection header received
0806 00:58:19.369439 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:19.369440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:19.369460 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:19.369551 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:19.371424 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:19.372325 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:19.374087 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:19.374315 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:19.374415 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:19.376905 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:19.376906 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:19.376906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:19.376913 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:19.378935 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:19.379485 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:19.379486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:19.379487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:19.379558 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:19.379832 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:19.379833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:19.379834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:19.379958 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:19.380033 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:19.380353 (+ 320us) server_negotiation.cc:299] Negotiation successful
0806 00:58:19.380447 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 00:58:19.380756 30487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:19.367621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:42293 (local address 127.23.242.1:46813)
0806 00:58:19.367830 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:19.367875 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:19.369025 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:19.369991 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:19.369997 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:19.370797 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:19.371275 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:19.371284 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:19.372465 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:19.372466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:19.373956 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:19.373959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:19.374430 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:19.374431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:19.374562 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:19.375316 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:19.375325 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:19.376778 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:19.379064 (+ 2286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:19.379072 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:19.379074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:19.379390 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:19.379652 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:19.379653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:19.379653 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:19.379765 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:19.380035 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:19.380043 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:19.380209 (+ 166us) client_negotiation.cc:769] Sending connection context
0806 00:58:19.380318 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 00:58:19.380484 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":103,"threads_started":1}
I20260806 00:58:19.381551 30485 heartbeater.cc:347] Connected to a master server at 127.23.242.62:42293
I20260806 00:58:19.381889 30485 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:19.382197 30485 heartbeater.cc:511] Master 127.23.242.62:42293 requested a full tablet report, sending...
I20260806 00:58:19.383056 30290 ts_manager.cc:194] Registered new tserver with Master: 628cdfe3d17246d1921f5720c9b905d2 (127.23.242.1:44785)
I20260806 00:58:19.383747 30290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.1@KRBTEST.COM'} at 127.23.242.1:46813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
I20260806 00:58:19.406782 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:0
--local_ip_for_outbound_sockets=127.23.242.2
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:42293
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:45675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:19.505622 30490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:19.505846 30490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:19.505873 30490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:19.507120 30490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:19.507169 30490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:19.507200 30490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 00:58:19.508666 30490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:45675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:42293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:19.508857 30490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:19.509038 30490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:19.514451 30496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:19.514741 30498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:19.514449 30495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:19.514804 30490 server_base.cc:1034] running on GCE node
I20260806 00:58:19.515350 30490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:19.515699 30490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:19.516921 30490 hybrid_clock.cc:648] HybridClock initialized: now 1785977899516903 us; error 33 us; skew 500 ppm
Aug 06 00:58:19 dist-test-slave-dhph krb5kdc[30244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977899, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:19.520255 30490 init.cc:377] Logged in from keytab as kudu/127.23.242.2@KRBTEST.COM (short username kudu)
I20260806 00:58:19.520946 30490 webserver.cc:460] Webserver started at http://127.23.242.2:40641/ using document root <none> and password file <none>
I20260806 00:58:19.521323 30490 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:19.521520 30490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:19.521772 30490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:19.523043 30490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-1/data/instance:
uuid: "8b6f192aabc44a48aad144de8686c893"
format_stamp: "Formatted at 2026-08-06 00:58:19 on dist-test-slave-dhph"
server_key: "eca1083b0348dd1cb167fc35578b4b60"
server_key_iv: "148a66d5680c1ff22b84a1df69367bbd"
server_key_version: "encryptionkey@0"
I20260806 00:58:19.523442 30490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance:
uuid: "8b6f192aabc44a48aad144de8686c893"
format_stamp: "Formatted at 2026-08-06 00:58:19 on dist-test-slave-dhph"
server_key: "eca1083b0348dd1cb167fc35578b4b60"
server_key_iv: "148a66d5680c1ff22b84a1df69367bbd"
server_key_version: "encryptionkey@0"
I20260806 00:58:19.525087 30490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260806 00:58:19.526278 30505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:19.526678 30490 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:19.526852 30490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "8b6f192aabc44a48aad144de8686c893"
format_stamp: "Formatted at 2026-08-06 00:58:19 on dist-test-slave-dhph"
server_key: "eca1083b0348dd1cb167fc35578b4b60"
server_key_iv: "148a66d5680c1ff22b84a1df69367bbd"
server_key_version: "encryptionkey@0"
I20260806 00:58:19.526970 30490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:19.541347 30490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:19.543941 30490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:19.544112 30490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:19.544377 30490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:19.544708 30490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:19.544765 30490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:19.544812 30490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:19.544845 30490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:19.550765 30490 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:44873
I20260806 00:58:19.550884 30620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:44873 every 8 connection(s)
I20260806 00:58:19.551203 30490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 00:58:19.554152 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 30490
I20260806 00:58:19.554265 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance
I20260806 00:58:19.554473 24520 external_mini_cluster.cc:1428] Setting key c68b22112962f7369b4dd61f7da1614a
Aug 06 00:58:19 dist-test-slave-dhph krb5kdc[30244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977899, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:19.563849 30341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:19.552205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:39235 (local address 127.23.242.62:42293)
0806 00:58:19.552278 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:19.552281 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:19.552293 (+ 12us) server_negotiation.cc:407] Connection header received
0806 00:58:19.553273 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:19.553273 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:19.553283 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:19.553338 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:19.553992 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:19.554837 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:19.556018 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:19.556372 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:19.556451 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:19.559266 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:19.559268 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:19.559268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:19.559277 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:19.561689 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:19.562397 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:19.562399 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:19.562400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:19.562522 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:19.563058 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:19.563059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:19.563059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:19.563203 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:19.563288 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:19.563691 (+ 403us) server_negotiation.cc:299] Negotiation successful
0806 00:58:19.563776 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 00:58:19.564117 30623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:19.551957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:42293 (local address 127.23.242.2:39235)
0806 00:58:19.552174 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:19.552225 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:19.553168 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:19.553395 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:19.553398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:19.553680 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:19.553880 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:19.553885 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:19.554965 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:19.554967 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:19.555927 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:19.555931 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:19.556494 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:19.556495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:19.556612 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:19.557252 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:19.557261 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:19.559145 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:19.561800 (+ 2655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:19.561811 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:19.561814 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:19.562219 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:19.562739 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:19.562741 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:19.562741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:19.562907 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:19.563294 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:19.563303 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:19.563496 (+ 193us) client_negotiation.cc:769] Sending connection context
0806 00:58:19.563630 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 00:58:19.563833 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":78,"threads_started":1}
I20260806 00:58:19.564808 30621 heartbeater.cc:347] Connected to a master server at 127.23.242.62:42293
I20260806 00:58:19.564963 30621 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.242.3@KRBTEST.COM; defaulting to no policy
I20260806 00:58:19.565245 30621 heartbeater.cc:511] Master 127.23.242.62:42293 requested a full tablet report, sending...
I20260806 00:58:19.565935 30290 ts_manager.cc:194] Registered new tserver with Master: 8b6f192aabc44a48aad144de8686c893 (127.23.242.2:44873)
I20260806 00:58:19.566455 30290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.2@KRBTEST.COM'} at 127.23.242.2:39235
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
I20260806 00:58:19.589699 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:0
--local_ip_for_outbound_sockets=127.23.242.3
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:42293
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:45675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:19.679106 30626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:19.679329 30626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:19.679356 30626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:19.680583 30626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:19.680632 30626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:19.680663 30626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 00:58:19.682233 30626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:45675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:42293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:19.682507 30626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:19.682844 30626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:19.689405 30631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:19.689703 30632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:19.689720 30634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:19.689924 30626 server_base.cc:1034] running on GCE node
I20260806 00:58:19.690130 30626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:19.690428 30626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:19.691632 30626 hybrid_clock.cc:648] HybridClock initialized: now 1785977899691601 us; error 49 us; skew 500 ppm
Aug 06 00:58:19 dist-test-slave-dhph krb5kdc[30244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977899, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:19.694675 30626 init.cc:377] Logged in from keytab as kudu/127.23.242.3@KRBTEST.COM (short username kudu)
I20260806 00:58:19.695340 30626 webserver.cc:460] Webserver started at http://127.23.242.3:43607/ using document root <none> and password file <none>
I20260806 00:58:19.695730 30626 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:19.695828 30626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:19.695999 30626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:19.697180 30626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-2/data/instance:
uuid: "ca673069e39e4b6199758b536ad8c33b"
format_stamp: "Formatted at 2026-08-06 00:58:19 on dist-test-slave-dhph"
server_key: "b54aa0110d4c36f91e5b26c712196631"
server_key_iv: "6dddb3ad405635687898908fe1ce501a"
server_key_version: "encryptionkey@0"
I20260806 00:58:19.697631 30626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance:
uuid: "ca673069e39e4b6199758b536ad8c33b"
format_stamp: "Formatted at 2026-08-06 00:58:19 on dist-test-slave-dhph"
server_key: "b54aa0110d4c36f91e5b26c712196631"
server_key_iv: "6dddb3ad405635687898908fe1ce501a"
server_key_version: "encryptionkey@0"
I20260806 00:58:19.699137 30626 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 00:58:19.700320 30641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:19.700642 30626 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:19.700785 30626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "ca673069e39e4b6199758b536ad8c33b"
format_stamp: "Formatted at 2026-08-06 00:58:19 on dist-test-slave-dhph"
server_key: "b54aa0110d4c36f91e5b26c712196631"
server_key_iv: "6dddb3ad405635687898908fe1ce501a"
server_key_version: "encryptionkey@0"
I20260806 00:58:19.700871 30626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:19.708235 30626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:19.710651 30626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:19.710830 30626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:19.711071 30626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:19.711428 30626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:19.711484 30626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:19.711521 30626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:19.711570 30626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:19.718958 30626 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:46437
I20260806 00:58:19.719198 30756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:46437 every 8 connection(s)
I20260806 00:58:19.719416 30626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
I20260806 00:58:19.725998 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 30626
I20260806 00:58:19.726102 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance
I20260806 00:58:19.726359 24520 external_mini_cluster.cc:1428] Setting key 9f608a3b27661cd334710ced38334c1b
Aug 06 00:58:19 dist-test-slave-dhph krb5kdc[30244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977899, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:19.731791 30341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:19.720261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:39489 (local address 127.23.242.62:42293)
0806 00:58:19.720314 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:19.720316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:19.720648 (+ 332us) server_negotiation.cc:407] Connection header received
0806 00:58:19.721631 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:19.721632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:19.721644 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:19.721710 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:19.722447 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:19.723129 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:19.724481 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:19.724676 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:19.724777 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:19.727914 (+ 3137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:19.727916 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:19.727916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:19.727923 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:19.729937 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:19.730511 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:19.730512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:19.730512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:19.730572 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:19.730967 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:19.730968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:19.730969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:19.731122 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:19.731192 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:19.731590 (+ 398us) server_negotiation.cc:299] Negotiation successful
0806 00:58:19.731676 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 00:58:19.731841 30759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:19.720269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:42293 (local address 127.23.242.3:39489)
0806 00:58:19.720482 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:19.720522 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:19.721467 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:19.721830 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:19.721834 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:19.722104 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:19.722319 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:19.722323 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:19.723317 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:19.723320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:19.724380 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:19.724385 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:19.724847 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:19.724848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:19.724970 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:19.725659 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:19.725669 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:19.727800 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:19.730058 (+ 2258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:19.730066 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:19.730068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:19.730421 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:19.730753 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:19.730754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:19.730755 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:19.730877 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:19.731215 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:19.731222 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:19.731355 (+ 133us) client_negotiation.cc:769] Sending connection context
0806 00:58:19.731467 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 00:58:19.731651 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":86,"threads_started":1}
I20260806 00:58:19.732740 30757 heartbeater.cc:347] Connected to a master server at 127.23.242.62:42293
I20260806 00:58:19.732892 30757 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:19.733119 30757 heartbeater.cc:511] Master 127.23.242.62:42293 requested a full tablet report, sending...
I20260806 00:58:19.733637 30290 ts_manager.cc:194] Registered new tserver with Master: ca673069e39e4b6199758b536ad8c33b (127.23.242.3:46437)
I20260806 00:58:19.734087 30290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.3@KRBTEST.COM'} at 127.23.242.3:39489
I20260806 00:58:19.739233 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/test-admin.keytab.
Aug 06 00:58:19 dist-test-slave-dhph krb5kdc[30244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:19.764266 24520 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 00:58:19.766665 30764 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:19.765355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41880 (local address 127.23.242.1:44785)
0806 00:58:19.765661 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:19.765667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:19.765687 (+ 20us) server_negotiation.cc:407] Connection header received
0806 00:58:19.765727 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:19.765729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:19.765750 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:19.765883 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:19.766451 (+ 568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41880: BlockingRecv error: recv got EOF from 127.0.0.1:41880 (error 108)
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":112,"threads_started":1}
W20260806 00:58:19.766685 24520 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/krb5cc' of type FILE
Aug 06 00:58:19 dist-test-slave-dhph krb5kdc[30244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:19.768229 24520 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 00:58:19 dist-test-slave-dhph krb5kdc[30244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:19.769272 24520 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 00:58:19 dist-test-slave-dhph krb5kdc[30244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.1@KRBTEST.COM
I20260806 00:58:19.780201 30764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:19.769929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41886 (local address 127.23.242.1:44785)
0806 00:58:19.770037 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:19.770039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:19.770051 (+ 12us) server_negotiation.cc:407] Connection header received
0806 00:58:19.770085 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:19.770086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:19.770101 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:19.770162 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:19.770669 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:19.771440 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:19.772520 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:19.773001 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:19.773117 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:19.776105 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:19.776120 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:19.776121 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:19.776131 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:19.778257 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:19.778794 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:19.778798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:19.778798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:19.778868 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:19.779235 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:19.779236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:19.779237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:19.779475 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:19.779929 (+ 454us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:19.780021 (+ 92us) server_negotiation.cc:299] Negotiation successful
0806 00:58:19.780105 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260806 00:58:19.782637 30764 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:19.781943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41902 (local address 127.23.242.1:44785)
0806 00:58:19.782011 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:19.782014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:19.782082 (+ 68us) server_negotiation.cc:407] Connection header received
0806 00:58:19.782182 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:19.782183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:19.782196 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:19.782252 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:19.782564 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41902: BlockingRecv error: recv got EOF from 127.0.0.1:41902 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260806 00:58:19.782956 24520 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/krb5cc' of type FILE
Aug 06 00:58:19 dist-test-slave-dhph krb5kdc[30244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:19.784087 24520 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 00:58:19 dist-test-slave-dhph krb5kdc[30244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:19.785068 24520 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 00:58:19 dist-test-slave-dhph krb5kdc[30244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.1@KRBTEST.COM
I20260806 00:58:19.793569 30764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:19.785686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41916 (local address 127.23.242.1:44785)
0806 00:58:19.785756 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:19.785758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:19.785769 (+ 11us) server_negotiation.cc:407] Connection header received
0806 00:58:19.785884 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:19.785885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:19.785904 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:19.785969 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:19.786442 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:19.787024 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:19.787785 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:19.787995 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:19.788099 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:19.790435 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:19.790436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:19.790436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:19.790443 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:19.791975 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:19.792385 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:19.792386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:19.792387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:19.792444 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:19.792954 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:19.792955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:19.792955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:19.793084 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:19.793179 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:19.793363 (+ 184us) server_negotiation.cc:299] Negotiation successful
0806 00:58:19.793450 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 00:58:19.795794 30764 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:19.794936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41932 (local address 127.23.242.1:44785)
0806 00:58:19.795002 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:19.795004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:19.795065 (+ 61us) server_negotiation.cc:407] Connection header received
0806 00:58:19.795146 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:19.795147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:19.795158 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:19.795237 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:19.795688 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41932: BlockingRecv error: recv got EOF from 127.0.0.1:41932 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260806 00:58:19.795935 24520 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785977869987768-24520-0/krb5kdc/krb5cc' of type FILE
Aug 06 00:58:19 dist-test-slave-dhph krb5kdc[30244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:19.796990 24520 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 00:58:19 dist-test-slave-dhph krb5kdc[30244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:19.798014 24520 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 00:58:19 dist-test-slave-dhph krb5kdc[30244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.1@KRBTEST.COM
I20260806 00:58:19.806221 30764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:19.798648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41936 (local address 127.23.242.1:44785)
0806 00:58:19.798751 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:19.798753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:19.798762 (+ 9us) server_negotiation.cc:407] Connection header received
0806 00:58:19.798806 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:19.798807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:19.798816 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:19.798868 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:19.799263 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:19.799803 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:19.800613 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:19.800867 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:19.800982 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:19.803371 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:19.803373 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:19.803373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:19.803425 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:19.805092 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:19.805491 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:19.805492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:19.805492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:19.805548 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:19.805737 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:19.805738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:19.805738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:19.805841 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:19.805928 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:19.806031 (+ 103us) server_negotiation.cc:299] Negotiation successful
0806 00:58:19.806094 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 00:58:19.807034 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 30354
I20260806 00:58:19.813875 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 30490
I20260806 00:58:19.821895 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 30626
I20260806 00:58:19.828851 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 30258
2026-08-06T00:58:19Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2147 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 00:58:19 dist-test-slave-dhph krb5kdc[30777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 00:58:19 dist-test-slave-dhph krb5kdc[30777](info): set up 2 sockets
Aug 06 00:58:19 dist-test-slave-dhph krb5kdc[30777](info): commencing operation
krb5kdc: starting...
W20260806 00:58:20.970816 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 00:58:21 dist-test-slave-dhph krb5kdc[30777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T00:58:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T00:58:21Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.62.keytab.
Entry for principal oryx/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.62.keytab.
I20260806 00:58:21.111207 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:38515
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.62.keytab
--principal=oryx/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:38515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:21.198874 30790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:21.199105 30790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:21.199131 30790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:21.200490 30790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 00:58:21.200553 30790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:21.200569 30790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:21.200582 30790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 00:58:21.200595 30790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 00:58:21.202747 30790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:38515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.62.keytab
--principal=oryx/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:38515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:21.202982 30790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:21.203176 30790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:21.209020 30798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:21.209020 30795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:21.209033 30796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:21.209477 30790 server_base.cc:1034] running on GCE node
I20260806 00:58:21.209765 30790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:21.210107 30790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:21.211396 30790 hybrid_clock.cc:648] HybridClock initialized: now 1785977901211379 us; error 36 us; skew 500 ppm
Aug 06 00:58:21 dist-test-slave-dhph krb5kdc[30777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977901, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:21.214417 30790 init.cc:377] Logged in from keytab as oryx/127.23.242.62@KRBTEST.COM (short username oryx)
I20260806 00:58:21.215085 30790 webserver.cc:460] Webserver started at http://127.23.242.62:36343/ using document root <none> and password file <none>
I20260806 00:58:21.215360 30790 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:21.215411 30790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:21.215552 30790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:21.216668 30790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "ee13c8d77d44483c8dec48001f37be48"
format_stamp: "Formatted at 2026-08-06 00:58:21 on dist-test-slave-dhph"
server_key: "a41d7822e7486695d8e79b07719951e5"
server_key_iv: "50009b0bf2b4bdd7985c5a6ee9a80800"
server_key_version: "encryptionkey@0"
I20260806 00:58:21.217355 30790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "ee13c8d77d44483c8dec48001f37be48"
format_stamp: "Formatted at 2026-08-06 00:58:21 on dist-test-slave-dhph"
server_key: "a41d7822e7486695d8e79b07719951e5"
server_key_iv: "50009b0bf2b4bdd7985c5a6ee9a80800"
server_key_version: "encryptionkey@0"
I20260806 00:58:21.219002 30790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260806 00:58:21.219920 30805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:21.220126 30790 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:21.220245 30790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "ee13c8d77d44483c8dec48001f37be48"
format_stamp: "Formatted at 2026-08-06 00:58:21 on dist-test-slave-dhph"
server_key: "a41d7822e7486695d8e79b07719951e5"
server_key_iv: "50009b0bf2b4bdd7985c5a6ee9a80800"
server_key_version: "encryptionkey@0"
I20260806 00:58:21.220299 30790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:21.242030 30790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:21.245069 30790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:21.245208 30790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:21.249791 30790 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:38515
I20260806 00:58:21.249910 30860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:38515 every 8 connection(s)
I20260806 00:58:21.250275 30790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 00:58:21.251628 30861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:21.254364 30861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48: Bootstrap starting.
I20260806 00:58:21.255149 30861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:21.255431 30861 log.cc:824] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:21.256371 30861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48: No bootstrap required, opened a new log
I20260806 00:58:21.257893 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 30790
I20260806 00:58:21.258003 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 00:58:21.258231 24520 external_mini_cluster.cc:1428] Setting key 8e375208cd624cbff2cdb12d5bb37bcf
I20260806 00:58:21.258147 30861 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } }
I20260806 00:58:21.258324 30861 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:21.258391 30861 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee13c8d77d44483c8dec48001f37be48, State: Initialized, Role: FOLLOWER
I20260806 00:58:21.258548 30861 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } }
I20260806 00:58:21.258664 30861 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 00:58:21.258724 30861 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 00:58:21.258785 30861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:21.259491 30861 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } }
I20260806 00:58:21.259618 30861 leader_election.cc:302] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee13c8d77d44483c8dec48001f37be48; no voters:
I20260806 00:58:21.259820 30861 leader_election.cc:288] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 00:58:21.259873 30866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:21.260089 30866 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [term 1 LEADER]: Becoming Leader. State: Replica: ee13c8d77d44483c8dec48001f37be48, State: Running, Role: LEADER
I20260806 00:58:21.260166 30861 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 00:58:21.260224 30866 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } }
I20260806 00:58:21.262018 30867 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee13c8d77d44483c8dec48001f37be48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } }
I20260806 00:58:21.262168 30867 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [sys.catalog]: This master's current role is: LEADER
W20260806 00:58:21.262470 30881 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 00:58:21.262557 30881 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 00:58:21.262441 30868 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee13c8d77d44483c8dec48001f37be48. Latest consensus state: current_term: 1 leader_uuid: "ee13c8d77d44483c8dec48001f37be48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } }
I20260806 00:58:21.262701 30883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 00:58:21.262755 30868 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:21.262948 30883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 00:58:21.264734 30883 catalog_manager.cc:1347] Generated new cluster ID: 3b0f30832a7d4834950ee317a3e241e2
I20260806 00:58:21.264786 30883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 00:58:21 dist-test-slave-dhph krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM
I20260806 00:58:21.270857 30865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:21.258838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57532 (local address 127.23.242.62:38515)
0806 00:58:21.259067 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:21.259077 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:21.259116 (+ 39us) server_negotiation.cc:407] Connection header received
0806 00:58:21.259488 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:21.259496 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:21.259546 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:21.259734 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:21.260550 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:21.261901 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:21.263143 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:21.263726 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:21.263844 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:21.266788 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:21.266792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:21.266792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:21.266804 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:21.269223 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:21.269604 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:21.269614 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:21.269615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:21.269720 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:21.269937 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:21.269937 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:21.269938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:21.270186 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:21.270351 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:21.270458 (+ 107us) server_negotiation.cc:299] Negotiation successful
0806 00:58:21.270643 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":135,"threads_started":1}
I20260806 00:58:21.283226 30883 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 00:58:21.283969 30883 catalog_manager.cc:1504] Loading token signing keys...
I20260806 00:58:21.293741 30883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48: Generated new TSK 0
I20260806 00:58:21.293992 30883 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.242.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.242.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.242.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.1.keytab.
Entry for principal oryx/127.23.242.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.1.keytab.
I20260806 00:58:21.331775 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:0
--local_ip_for_outbound_sockets=127.23.242.1
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:38515
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.1.keytab
--principal=oryx/127.23.242.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:21.424428 30887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:21.424664 30887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:21.424695 30887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:21.426128 30887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:21.426201 30887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:21.426231 30887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 00:58:21.427891 30887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.1.keytab
--principal=oryx/127.23.242.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:38515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:21.428130 30887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:21.428349 30887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:21.433892 30895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:21.433880 30893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:21.433940 30887 server_base.cc:1034] running on GCE node
W20260806 00:58:21.433877 30892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:21.434391 30887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:21.434716 30887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:21.435870 30887 hybrid_clock.cc:648] HybridClock initialized: now 1785977901435851 us; error 36 us; skew 500 ppm
Aug 06 00:58:21 dist-test-slave-dhph krb5kdc[30777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977901, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:21.438920 30887 init.cc:377] Logged in from keytab as oryx/127.23.242.1@KRBTEST.COM (short username oryx)
I20260806 00:58:21.439579 30887 webserver.cc:460] Webserver started at http://127.23.242.1:33405/ using document root <none> and password file <none>
I20260806 00:58:21.439855 30887 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:21.439934 30887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:21.440101 30887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:21.441324 30887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/data/instance:
uuid: "ba9ec1e1be664d3b9e705abaf8f50594"
format_stamp: "Formatted at 2026-08-06 00:58:21 on dist-test-slave-dhph"
server_key: "bf11ceecb7960f30dfa81fd30af85064"
server_key_iv: "1bfb90eff4e80ce3a0528a0fd97d9b9a"
server_key_version: "encryptionkey@0"
I20260806 00:58:21.441737 30887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance:
uuid: "ba9ec1e1be664d3b9e705abaf8f50594"
format_stamp: "Formatted at 2026-08-06 00:58:21 on dist-test-slave-dhph"
server_key: "bf11ceecb7960f30dfa81fd30af85064"
server_key_iv: "1bfb90eff4e80ce3a0528a0fd97d9b9a"
server_key_version: "encryptionkey@0"
I20260806 00:58:21.443372 30887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 00:58:21.444348 30902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:21.444623 30887 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:21.444756 30887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "ba9ec1e1be664d3b9e705abaf8f50594"
format_stamp: "Formatted at 2026-08-06 00:58:21 on dist-test-slave-dhph"
server_key: "bf11ceecb7960f30dfa81fd30af85064"
server_key_iv: "1bfb90eff4e80ce3a0528a0fd97d9b9a"
server_key_version: "encryptionkey@0"
I20260806 00:58:21.444839 30887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:21.453444 30887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:21.456259 30887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:21.456449 30887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:21.456743 30887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:21.457122 30887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:21.457191 30887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:21.457235 30887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:21.457265 30887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:21.465166 30887 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:42435
I20260806 00:58:21.465227 31018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:42435 every 8 connection(s)
I20260806 00:58:21.465675 30887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 00:58:21.468376 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 30887
I20260806 00:58:21.468534 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance
I20260806 00:58:21.468778 24520 external_mini_cluster.cc:1428] Setting key 953be4c69dbc251af58235f920d27a4e
Aug 06 00:58:21 dist-test-slave-dhph krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977901, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM
I20260806 00:58:21.476904 30865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:21.466541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:34323 (local address 127.23.242.62:38515)
0806 00:58:21.466636 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:21.466639 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:21.466880 (+ 241us) server_negotiation.cc:407] Connection header received
0806 00:58:21.467418 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:21.467420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:21.467436 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:21.467516 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:21.468533 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:21.469284 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:21.470367 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:21.470691 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:21.470770 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:21.473112 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:21.473114 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:21.473114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:21.473122 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:21.475137 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:21.475698 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:21.475700 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:21.475700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:21.475770 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:21.476090 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:21.476091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:21.476091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:21.476268 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:21.476351 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:21.476645 (+ 294us) server_negotiation.cc:299] Negotiation successful
0806 00:58:21.476762 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 00:58:21.476923 31021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:21.466523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.23.242.1:34323)
0806 00:58:21.466744 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:21.466794 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:21.467326 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:21.467574 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:21.467578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:21.468188 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:21.468419 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:21.468425 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:21.469404 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:21.469406 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:21.470267 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:21.470271 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:21.470781 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:21.470782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:21.470924 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:21.471510 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:21.471518 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:21.473007 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:21.475246 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:21.475253 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:21.475256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:21.475587 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:21.475880 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:21.475881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:21.475881 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:21.476000 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:21.476351 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:21.476359 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:21.476498 (+ 139us) client_negotiation.cc:769] Sending connection context
0806 00:58:21.476601 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 00:58:21.476743 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":115,"threads_started":1}
I20260806 00:58:21.477547 31019 heartbeater.cc:347] Connected to a master server at 127.23.242.62:38515
I20260806 00:58:21.477658 31019 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:21.477975 31019 heartbeater.cc:511] Master 127.23.242.62:38515 requested a full tablet report, sending...
I20260806 00:58:21.478787 30822 ts_manager.cc:194] Registered new tserver with Master: ba9ec1e1be664d3b9e705abaf8f50594 (127.23.242.1:42435)
I20260806 00:58:21.479502 30822 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.242.1@KRBTEST.COM'} at 127.23.242.1:34323
WARNING: no policy specified for oryx/127.23.242.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.242.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.242.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.2.keytab.
Entry for principal oryx/127.23.242.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.2.keytab.
I20260806 00:58:21.503625 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:0
--local_ip_for_outbound_sockets=127.23.242.2
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:38515
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.2.keytab
--principal=oryx/127.23.242.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:21.595904 31024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:21.596141 31024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:21.596169 31024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:21.597411 31024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:21.597460 31024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:21.597487 31024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 00:58:21.599807 31024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.2.keytab
--principal=oryx/127.23.242.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:38515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:21.600024 31024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:21.600229 31024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:21.606508 31032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:21.606837 31024 server_base.cc:1034] running on GCE node
W20260806 00:58:21.606511 31030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:21.606508 31029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:21.607195 31024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:21.607604 31024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:21.608816 31024 hybrid_clock.cc:648] HybridClock initialized: now 1785977901608775 us; error 72 us; skew 500 ppm
Aug 06 00:58:21 dist-test-slave-dhph krb5kdc[30777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977901, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:21.612311 31024 init.cc:377] Logged in from keytab as oryx/127.23.242.2@KRBTEST.COM (short username oryx)
I20260806 00:58:21.613085 31024 webserver.cc:460] Webserver started at http://127.23.242.2:35943/ using document root <none> and password file <none>
I20260806 00:58:21.613394 31024 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:21.613480 31024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:21.613662 31024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:21.615002 31024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/data/instance:
uuid: "4765fb0b489240bb920242b63a57e71f"
format_stamp: "Formatted at 2026-08-06 00:58:21 on dist-test-slave-dhph"
server_key: "234cd73e37450275ff372942ddfca717"
server_key_iv: "6c71b11e8cdda8b72761829a2c0dd594"
server_key_version: "encryptionkey@0"
I20260806 00:58:21.615437 31024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance:
uuid: "4765fb0b489240bb920242b63a57e71f"
format_stamp: "Formatted at 2026-08-06 00:58:21 on dist-test-slave-dhph"
server_key: "234cd73e37450275ff372942ddfca717"
server_key_iv: "6c71b11e8cdda8b72761829a2c0dd594"
server_key_version: "encryptionkey@0"
I20260806 00:58:21.617097 31024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 00:58:21.618129 31039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:21.618410 31024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:21.618536 31024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "4765fb0b489240bb920242b63a57e71f"
format_stamp: "Formatted at 2026-08-06 00:58:21 on dist-test-slave-dhph"
server_key: "234cd73e37450275ff372942ddfca717"
server_key_iv: "6c71b11e8cdda8b72761829a2c0dd594"
server_key_version: "encryptionkey@0"
I20260806 00:58:21.618726 31024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:21.634258 31024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:21.637805 31024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:21.638028 31024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:21.638360 31024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:21.638968 31024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:21.639045 31024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:21.639103 31024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:21.639161 31024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:21.646842 31024 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:45035
I20260806 00:58:21.646942 31154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:45035 every 8 connection(s)
I20260806 00:58:21.647326 31024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 00:58:21.650638 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 31024
I20260806 00:58:21.650799 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance
I20260806 00:58:21.651041 24520 external_mini_cluster.cc:1428] Setting key 0966fd141d6f285fd51d0368f7d68d3d
Aug 06 00:58:21 dist-test-slave-dhph krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977901, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM
I20260806 00:58:21.660245 30865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:21.648263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:45307 (local address 127.23.242.62:38515)
0806 00:58:21.648368 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:21.648371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:21.648536 (+ 165us) server_negotiation.cc:407] Connection header received
0806 00:58:21.649307 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:21.649308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:21.649324 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:21.649383 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:21.650942 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:21.651694 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:21.653865 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:21.654109 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:21.654217 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:21.656719 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:21.656721 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:21.656721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:21.656730 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:21.658294 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:21.658886 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:21.658887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:21.658887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:21.658958 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:21.659247 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:21.659248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:21.659249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:21.659487 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:21.659621 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:21.660028 (+ 407us) server_negotiation.cc:299] Negotiation successful
0806 00:58:21.660133 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 00:58:21.660573 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:21.648133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.23.242.2:45307)
0806 00:58:21.648381 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:21.648440 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:21.649168 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:21.649465 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:21.649469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:21.650311 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:21.650815 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:21.650823 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:21.651893 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:21.651915 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:21.653739 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:21.653746 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:21.654221 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:21.654223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:21.654402 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:21.655125 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:21.655133 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:21.656598 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:21.658410 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:21.658419 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:21.658422 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:21.658801 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:21.659033 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:21.659035 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:21.659035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:21.659166 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:21.659606 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:21.659621 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:21.659855 (+ 234us) client_negotiation.cc:769] Sending connection context
0806 00:58:21.659987 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 00:58:21.660205 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":117,"threads_started":1}
I20260806 00:58:21.661178 31155 heartbeater.cc:347] Connected to a master server at 127.23.242.62:38515
I20260806 00:58:21.661314 31155 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:21.661538 31155 heartbeater.cc:511] Master 127.23.242.62:38515 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.23.242.3@KRBTEST.COM; defaulting to no policy
I20260806 00:58:21.661993 30822 ts_manager.cc:194] Registered new tserver with Master: 4765fb0b489240bb920242b63a57e71f (127.23.242.2:45035)
I20260806 00:58:21.662429 30822 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.242.2@KRBTEST.COM'} at 127.23.242.2:45307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.242.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.242.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.3.keytab.
Entry for principal oryx/127.23.242.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.3.keytab.
I20260806 00:58:21.687085 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:0
--local_ip_for_outbound_sockets=127.23.242.3
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:38515
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.3.keytab
--principal=oryx/127.23.242.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:21.782790 31160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:21.783052 31160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:21.783093 31160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:21.784358 31160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:21.784436 31160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:21.784483 31160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 00:58:21.786319 31160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.3.keytab
--principal=oryx/127.23.242.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:38515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:21.786706 31160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:21.787026 31160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 00:58:21.793670 31160 server_base.cc:1034] running on GCE node
W20260806 00:58:21.793627 31165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:21.793736 31168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:21.793586 31166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:21.794239 31160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:21.794646 31160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:21.795861 31160 hybrid_clock.cc:648] HybridClock initialized: now 1785977901795810 us; error 78 us; skew 500 ppm
Aug 06 00:58:21 dist-test-slave-dhph krb5kdc[30777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977901, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:21.798960 31160 init.cc:377] Logged in from keytab as oryx/127.23.242.3@KRBTEST.COM (short username oryx)
I20260806 00:58:21.799467 31160 webserver.cc:460] Webserver started at http://127.23.242.3:44667/ using document root <none> and password file <none>
I20260806 00:58:21.799922 31160 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:21.800015 31160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:21.800184 31160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:21.801277 31160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/data/instance:
uuid: "ec07cecb7fe24fed939834282411206d"
format_stamp: "Formatted at 2026-08-06 00:58:21 on dist-test-slave-dhph"
server_key: "055df28b4db838a8f4ef96f5ec85a6b1"
server_key_iv: "39259a08e3a56df0766146aa94e095c3"
server_key_version: "encryptionkey@0"
I20260806 00:58:21.801716 31160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance:
uuid: "ec07cecb7fe24fed939834282411206d"
format_stamp: "Formatted at 2026-08-06 00:58:21 on dist-test-slave-dhph"
server_key: "055df28b4db838a8f4ef96f5ec85a6b1"
server_key_iv: "39259a08e3a56df0766146aa94e095c3"
server_key_version: "encryptionkey@0"
I20260806 00:58:21.803421 31160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260806 00:58:21.804608 31175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:21.804961 31160 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:21.805106 31160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "ec07cecb7fe24fed939834282411206d"
format_stamp: "Formatted at 2026-08-06 00:58:21 on dist-test-slave-dhph"
server_key: "055df28b4db838a8f4ef96f5ec85a6b1"
server_key_iv: "39259a08e3a56df0766146aa94e095c3"
server_key_version: "encryptionkey@0"
I20260806 00:58:21.805208 31160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:21.824102 31160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:21.826486 31160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:21.826840 31160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:21.827147 31160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:21.827495 31160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:21.827554 31160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:21.827595 31160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:21.827644 31160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:21.833855 31160 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:39943
I20260806 00:58:21.833899 31290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:39943 every 8 connection(s)
I20260806 00:58:21.834257 31160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
Aug 06 00:58:21 dist-test-slave-dhph krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977901, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM
I20260806 00:58:21.843802 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 31160
I20260806 00:58:21.843899 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance
I20260806 00:58:21.844123 24520 external_mini_cluster.cc:1428] Setting key 2f77d8a167921282dec5bcdfc6af8c9b
I20260806 00:58:21.846421 30865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:21.835136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:59049 (local address 127.23.242.62:38515)
0806 00:58:21.835216 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:21.835219 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:21.835571 (+ 352us) server_negotiation.cc:407] Connection header received
0806 00:58:21.836057 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:21.836059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:21.836070 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:21.836136 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:21.837667 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:21.838197 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:21.839414 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:21.839614 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:21.839697 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:21.842655 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:21.842658 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:21.842659 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:21.842669 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:21.844765 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:21.845388 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:21.845389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:21.845390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:21.845467 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:21.845715 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:21.845716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:21.845716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:21.845847 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:21.845908 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:21.846245 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 00:58:21.846347 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 00:58:21.846534 31293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:21.835104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.23.242.3:59049)
0806 00:58:21.835365 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:21.835404 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:21.835924 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:21.836467 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:21.836471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:21.837290 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:21.837516 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:21.837521 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:21.838300 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:21.838301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:21.839277 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:21.839280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:21.839732 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:21.839733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:21.839860 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:21.840454 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:21.840462 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:21.842272 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:21.844899 (+ 2627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:21.844912 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:21.844915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:21.845304 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:21.845535 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:21.845536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:21.845536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:21.845644 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:21.845913 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:21.845918 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:21.846046 (+ 128us) client_negotiation.cc:769] Sending connection context
0806 00:58:21.846134 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 00:58:21.846309 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":119,"threads_started":1}
I20260806 00:58:21.847141 31291 heartbeater.cc:347] Connected to a master server at 127.23.242.62:38515
I20260806 00:58:21.847294 31291 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:21.847605 31291 heartbeater.cc:511] Master 127.23.242.62:38515 requested a full tablet report, sending...
I20260806 00:58:21.848165 30822 ts_manager.cc:194] Registered new tserver with Master: ec07cecb7fe24fed939834282411206d (127.23.242.3:39943)
I20260806 00:58:21.848666 30822 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.242.3@KRBTEST.COM'} at 127.23.242.3:59049
I20260806 00:58:21.856884 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.242.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.242.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.242.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.61.keytab.
Entry for principal oryx/127.23.242.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.61.keytab.
I20260806 00:58:21.891402 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.61:41127
--webserver_interface=127.23.242.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.61.keytab
--principal=oryx/127.23.242.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:38515,127.23.242.61:41127
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:21.981418 31296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:21.981675 31296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:21.981721 31296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:21.982895 31296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 00:58:21.982964 31296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:21.982990 31296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:21.983018 31296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 00:58:21.983050 31296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 00:58:21.984869 31296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:38515,127.23.242.61:41127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.61.keytab
--principal=oryx/127.23.242.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.61:41127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.242.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
W20260806 00:58:21.985061 31296 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.23.242.62:38515,127.23.242.61:41127'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 00:58:21.985241 31296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:21.985512 31296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:21.991349 31302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:21.991509 31304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:21.991364 31301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:21.991799 31296 server_base.cc:1034] running on GCE node
I20260806 00:58:21.992120 31296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:21.992440 31296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:21.993664 31296 hybrid_clock.cc:648] HybridClock initialized: now 1785977901993657 us; error 40 us; skew 500 ppm
Aug 06 00:58:21 dist-test-slave-dhph krb5kdc[30777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977901, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:21.996703 31296 init.cc:377] Logged in from keytab as oryx/127.23.242.61@KRBTEST.COM (short username oryx)
I20260806 00:58:21.997505 31296 webserver.cc:460] Webserver started at http://127.23.242.61:43597/ using document root <none> and password file <none>
I20260806 00:58:21.997805 31296 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:21.997891 31296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:21.998081 31296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:21.999405 31296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-1/data/instance:
uuid: "a6ca6242d6744e37a9d2b1aa76faef3c"
format_stamp: "Formatted at 2026-08-06 00:58:21 on dist-test-slave-dhph"
server_key: "60480817314b3f3b238f4f1829597812"
server_key_iv: "e4d214bd06e8b6584605bfd2f8393de4"
server_key_version: "encryptionkey@0"
I20260806 00:58:21.999903 31296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-1/wal/instance:
uuid: "a6ca6242d6744e37a9d2b1aa76faef3c"
format_stamp: "Formatted at 2026-08-06 00:58:21 on dist-test-slave-dhph"
server_key: "60480817314b3f3b238f4f1829597812"
server_key_iv: "e4d214bd06e8b6584605bfd2f8393de4"
server_key_version: "encryptionkey@0"
I20260806 00:58:22.001792 31296 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 00:58:22.003273 31311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:22.003957 31296 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:22.004104 31296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-1/wal
uuid: "a6ca6242d6744e37a9d2b1aa76faef3c"
format_stamp: "Formatted at 2026-08-06 00:58:21 on dist-test-slave-dhph"
server_key: "60480817314b3f3b238f4f1829597812"
server_key_iv: "e4d214bd06e8b6584605bfd2f8393de4"
server_key_version: "encryptionkey@0"
I20260806 00:58:22.004220 31296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:22.025255 31296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:22.027904 31296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:22.028112 31296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:22.032843 31296 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.61:41127
I20260806 00:58:22.032881 31365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.61:41127 every 8 connection(s)
I20260806 00:58:22.033249 31296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-1/data/info.pb
I20260806 00:58:22.034533 31366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:22.036464 31366 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } has no permanent_uuid. Determining permanent_uuid...
I20260806 00:58:22.038184 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 31296
Aug 06 00:58:22 dist-test-slave-dhph krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977901, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.61@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM
Aug 06 00:58:22 dist-test-slave-dhph krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.242.61@KRBTEST.COM
I20260806 00:58:22.048954 30865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.037357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57534 (local address 127.23.242.62:38515)
0806 00:58:22.037449 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.037452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.037760 (+ 308us) server_negotiation.cc:407] Connection header received
0806 00:58:22.038706 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.038710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.038739 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.038988 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.039920 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.040669 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.042078 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.042373 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.042456 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.045076 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.045078 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.045078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.045094 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:22.046815 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.047579 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.047581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.047581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.047662 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.047955 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.047956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.047957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.048189 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:22.048297 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:22.048744 (+ 447us) server_negotiation.cc:299] Negotiation successful
0806 00:58:22.048841 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 00:58:22.049063 31368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.037296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.0.0.1:57534)
0806 00:58:22.037545 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:22.037585 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:22.038313 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:22.039211 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:22.039216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:22.039638 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:22.039799 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.039803 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.040763 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.040764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.041990 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.041992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.042652 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.042653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.042774 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.043472 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:22.043480 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:22.044960 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:22.046910 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.046918 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.046920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.047419 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.047744 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.047745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.047746 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.047864 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.048283 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:22.048291 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:22.048485 (+ 194us) client_negotiation.cc:769] Sending connection context
0806 00:58:22.048616 (+ 131us) client_negotiation.cc:240] Negotiation successful
0806 00:58:22.048805 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":113,"threads_started":1}
I20260806 00:58:22.049762 31370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.039090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33802 (local address 127.23.242.61:41127)
0806 00:58:22.039273 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.039278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.039321 (+ 43us) server_negotiation.cc:407] Connection header received
0806 00:58:22.039359 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.039361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.039389 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.039493 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.040347 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.041799 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.042832 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.043241 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.043347 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.046207 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.046216 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.046216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.046225 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:22.048277 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.048742 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.048745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.048746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.048803 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.049022 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.049023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.049023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.049213 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:22.049352 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:22.049554 (+ 202us) server_negotiation.cc:299] Negotiation successful
0806 00:58:22.049638 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":87,"threads_started":1}
I20260806 00:58:22.049737 31366 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 00:58:22 dist-test-slave-dhph krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977901, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.61@KRBTEST.COM for oryx/127.23.242.61@KRBTEST.COM
I20260806 00:58:22.057603 31368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.050155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:41127 (local address 127.0.0.1:33818)
0806 00:58:22.050220 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:22.050230 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:22.050279 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:22.050392 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:22.050392 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:22.050544 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:22.050606 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.050608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.051244 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.051245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.051881 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.051883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.052379 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.052380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.052428 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.052874 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:22.052879 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:22.054185 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:22.056192 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.056193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.056193 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.056385 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.056660 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.056661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.056661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.056763 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.057144 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:22.057145 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:22.057360 (+ 215us) client_negotiation.cc:769] Sending connection context
0806 00:58:22.057443 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 00:58:22.057526 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 00:58:22.057710 31370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.050183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33818 (local address 127.23.242.61:41127)
0806 00:58:22.050216 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.050218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.050271 (+ 53us) server_negotiation.cc:407] Connection header received
0806 00:58:22.050331 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.050332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.050340 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.050378 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.050679 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.051132 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.052036 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.052266 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.052331 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.054306 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.054308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.054308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.054315 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:22.056057 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.056493 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.056494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.056494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.056551 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.056852 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.056853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.056853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.057036 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:22.057129 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:22.057499 (+ 370us) server_negotiation.cc:299] Negotiation successful
0806 00:58:22.057587 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":16}
I20260806 00:58:22.059621 31366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c: Bootstrap starting.
I20260806 00:58:22.060770 31366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:22.061096 31366 log.cc:824] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:22.062044 31366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c: No bootstrap required, opened a new log
I20260806 00:58:22.063563 31366 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } }
I20260806 00:58:22.063689 31366 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:22.063711 31366 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6ca6242d6744e37a9d2b1aa76faef3c, State: Initialized, Role: FOLLOWER
I20260806 00:58:22.063846 31366 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } }
I20260806 00:58:22.064131 31372 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } } }
I20260806 00:58:22.064155 31366 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 00:58:22.064294 31372 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [sys.catalog]: This master's current role is: FOLLOWER
W20260806 00:58:22.066470 31383 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 00:58:22.066540 31383 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 00:58:22.068876 31296 master_runner.cc:428] Detected that this master a6ca6242d6744e37a9d2b1aa76faef3c is joining an existing cluster
I20260806 00:58:22.068955 31296 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.242.61:41127
I20260806 00:58:22.078315 30865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.070180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57540 (local address 127.23.242.62:38515)
0806 00:58:22.070246 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.070248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.070393 (+ 145us) server_negotiation.cc:407] Connection header received
0806 00:58:22.070516 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.070517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.070530 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.070633 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.071313 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.071826 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.072567 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.072788 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.072866 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.074070 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.074071 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.074072 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.074079 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:22.076071 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.077124 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.077126 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.077126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.077191 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.077658 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.077660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.077660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.077825 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:22.077927 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:22.078102 (+ 175us) server_negotiation.cc:299] Negotiation successful
0806 00:58:22.078219 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 00:58:22.078327 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.070142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.0.0.1:57540)
0806 00:58:22.070313 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:22.070326 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:22.070424 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:22.070635 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:22.070636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:22.071087 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:22.071155 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.071157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.071910 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.071911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.072478 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.072479 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.072874 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.072875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.072946 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.073510 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:22.073516 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:22.073976 (+ 460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:22.076780 (+ 2804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.076781 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.076781 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.077037 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.077294 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.077295 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.077295 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.077358 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.077928 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:22.077929 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:22.077975 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 00:58:22.078065 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 00:58:22.078191 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":104,"threads_started":1}
I20260806 00:58:22.078715 31370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.070418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33834 (local address 127.23.242.61:41127)
0806 00:58:22.070458 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.070460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.070469 (+ 9us) server_negotiation.cc:407] Connection header received
0806 00:58:22.070492 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.070493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.070503 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.070564 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.071415 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.072275 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.073399 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.073735 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.073820 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.075027 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.075028 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.075028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.075036 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:22.076697 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.077267 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.077268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.077269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.077329 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.078107 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.078108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.078108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.078228 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:22.078271 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:22.078446 (+ 175us) server_negotiation.cc:299] Negotiation successful
0806 00:58:22.078526 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 00:58:22.079035 31390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.070210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:41127 (local address 127.0.0.1:33834)
0806 00:58:22.070365 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:22.070375 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:22.070424 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:22.070651 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:22.070652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:22.071059 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:22.071156 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.071159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.072398 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.072399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.073272 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.073275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.073843 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.073845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.073918 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.074405 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:22.074411 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:22.074923 (+ 512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:22.076794 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.076795 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.076795 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.077185 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.077955 (+ 770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.077956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.077956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.078030 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.078314 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:22.078315 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:22.078347 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 00:58:22.078834 (+ 487us) client_negotiation.cc:240] Negotiation successful
0806 00:58:22.078932 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":2,"thread_start_us":113,"threads_started":1}
Aug 06 00:58:22 dist-test-slave-dhph krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977901, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.62@KRBTEST.COM for oryx/127.23.242.61@KRBTEST.COM
I20260806 00:58:22.091279 31370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.081685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33848 (local address 127.23.242.61:41127)
0806 00:58:22.081797 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.081799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.082827 (+ 1028us) server_negotiation.cc:407] Connection header received
0806 00:58:22.082914 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.082915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.082925 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.082986 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.083551 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.084033 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.085172 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.085388 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.085459 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.087926 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.087927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.087927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.087937 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:22.089749 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.090261 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.090262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.090263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.090334 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.090674 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.090675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.090676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.090808 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:22.090876 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:22.091135 (+ 259us) server_negotiation.cc:299] Negotiation successful
0806 00:58:22.091211 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 00:58:22.091369 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.081554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:41127 (local address 127.0.0.1:33848)
0806 00:58:22.082702 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:22.082753 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:22.082832 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:22.083010 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:22.083015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:22.083390 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:22.083455 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.083458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.084118 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.084119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.085071 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.085074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.085474 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.085474 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.085567 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.086053 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:22.086058 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:22.087784 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:22.089842 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.089852 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.089855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.090149 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.090437 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.090438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.090439 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.090529 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.090894 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:22.090903 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:22.090972 (+ 69us) client_negotiation.cc:769] Sending connection context
0806 00:58:22.091069 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 00:58:22.091221 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1115,"thread_start_us":83,"threads_started":1}
I20260806 00:58:22.092130 30822 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.242.61:41127: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c"
member_type: NON_VOTER
last_known_addr {
host: "127.23.242.61"
port: 41127
}
attrs {
promote: true
}
}
dest_uuid: "ee13c8d77d44483c8dec48001f37be48"
cas_config_opid_index: -1
I20260806 00:58:22.092501 30822 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: NON_VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: true } }
I20260806 00:58:22.093052 31393 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a6ca6242d6744e37a9d2b1aa76faef3c (127.23.242.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: NON_VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: true } } }
I20260806 00:58:22.093415 31339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:22.094213 31339 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 1 FOLLOWER]: Refusing update from remote peer ee13c8d77d44483c8dec48001f37be48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260806 00:58:22.094205 31393 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.242.61:41127
I20260806 00:58:22.094318 31393 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ee13c8d77d44483c8dec48001f37be48" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: NON_VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: true } } }
I20260806 00:58:22.094388 31393 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:22.094643 31296 master.cc:537] Master@127.23.242.61:41127 shutting down...
I20260806 00:58:22.094733 31392 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: NON_VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 00:58:22.095464 30808 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260806 00:58:22.096047 30808 consensus_peers.cc:487] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 -> Peer a6ca6242d6744e37a9d2b1aa76faef3c (127.23.242.61:41127): Couldn't send request to peer a6ca6242d6744e37a9d2b1aa76faef3c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 00:58:22.098390 31296 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:22.098527 31296 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:22.098613 31296 tablet_replica.cc:330] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c: stopping tablet replica
I20260806 00:58:22.103013 31296 master.cc:552] Master@127.23.242.61:41127 shutdown complete.
I20260806 00:58:22.103089 31296 master_runner.cc:305] Clearing existing system tablet
I20260806 00:58:22.103231 31296 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:22.103296 31296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:22.103988 31296 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260806 00:58:22.104524 31397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:22.104650 31296 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:22.104707 31296 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 00:58:22.104740 31296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-1/wal
uuid: "a6ca6242d6744e37a9d2b1aa76faef3c"
format_stamp: "Formatted at 2026-08-06 00:58:21 on dist-test-slave-dhph"
server_key: "60480817314b3f3b238f4f1829597812"
server_key_iv: "e4d214bd06e8b6584605bfd2f8393de4"
server_key_version: "encryptionkey@0"
I20260806 00:58:22.105437 31296 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:22.106997 31296 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 00:58:22.107120 31296 log.cc:1196] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260806 00:58:22.107398 31296 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c: Deleting consensus metadata
I20260806 00:58:22.107643 31296 master_runner.cc:315] Copying system tablet from 127.23.242.62:38515
I20260806 00:58:22.108400 31296 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c: tablet copy: Beginning tablet copy session from remote peer at address 127.23.242.62:38515
I20260806 00:58:22.114790 30865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.108807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57542 (local address 127.23.242.62:38515)
0806 00:58:22.108868 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.108870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.108976 (+ 106us) server_negotiation.cc:407] Connection header received
0806 00:58:22.109051 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.109052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.109064 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.109115 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.109455 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.110027 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.110822 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.111031 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.111118 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.112135 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.112136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.112137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.112143 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:22.113518 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.113878 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.113879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.113879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.113936 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.114155 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.114156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.114156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.114279 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:22.114351 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:22.114540 (+ 189us) server_negotiation.cc:299] Negotiation successful
0806 00:58:22.114692 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260806 00:58:22.114805 31403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.108729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.0.0.1:57542)
0806 00:58:22.108905 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:22.108914 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:22.108994 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:22.109117 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:22.109117 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:22.109319 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:22.109375 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.109377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.110114 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.110115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.110739 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.110741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.111119 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.111120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.111188 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.111706 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:22.111712 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:22.112022 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:22.113620 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.113621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.113621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.113803 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.114013 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.114014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.114014 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.114075 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.114354 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:22.114356 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:22.114413 (+ 57us) client_negotiation.cc:769] Sending connection context
0806 00:58:22.114497 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 00:58:22.114694 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":67,"threads_started":1}
I20260806 00:58:22.115336 30844 tablet_copy_service.cc:140] P ee13c8d77d44483c8dec48001f37be48: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a6ca6242d6744e37a9d2b1aa76faef3c ({username='oryx', principal='oryx/127.23.242.61@KRBTEST.COM'} at 127.0.0.1:57542)
I20260806 00:58:22.115453 30844 tablet_copy_service.cc:161] P ee13c8d77d44483c8dec48001f37be48: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a6ca6242d6744e37a9d2b1aa76faef3c at {username='oryx', principal='oryx/127.23.242.61@KRBTEST.COM'} at 127.0.0.1:57542: session id = a6ca6242d6744e37a9d2b1aa76faef3c-00000000000000000000000000000000
I20260806 00:58:22.116283 30844 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 00:58:22.116971 31296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 00:58:22.118218 31296 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c: tablet copy: Starting download of 0 data blocks...
I20260806 00:58:22.118413 31296 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c: tablet copy: Starting download of 1 WAL segments...
I20260806 00:58:22.119635 31296 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 00:58:22.120509 30844 tablet_copy_service.cc:342] P ee13c8d77d44483c8dec48001f37be48: Request end of tablet copy session a6ca6242d6744e37a9d2b1aa76faef3c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.242.61@KRBTEST.COM'} at 127.0.0.1:57542
I20260806 00:58:22.120581 30844 tablet_copy_service.cc:433] P ee13c8d77d44483c8dec48001f37be48: ending tablet copy session a6ca6242d6744e37a9d2b1aa76faef3c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a6ca6242d6744e37a9d2b1aa76faef3c
I20260806 00:58:22.124491 31296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:22.126292 31404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:22.126353 31405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:22.127678 31407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:22.127806 31296 server_base.cc:1034] running on GCE node
I20260806 00:58:22.127995 31296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:22.128154 31296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:22.129302 31296 hybrid_clock.cc:648] HybridClock initialized: now 1785977902129292 us; error 26 us; skew 500 ppm
Aug 06 00:58:22 dist-test-slave-dhph krb5kdc[30777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977902, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:22.130774 31296 init.cc:377] Logged in from keytab as oryx/127.23.242.61@KRBTEST.COM (short username oryx)
I20260806 00:58:22.131085 31296 webserver.cc:460] Webserver started at http://127.23.242.61:38429/ using document root <none> and password file <none>
I20260806 00:58:22.131222 31296 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:22.131273 31296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:22.131922 31296 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260806 00:58:22.132493 31414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:22.132614 31296 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 00:58:22.132661 31296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-1/wal
uuid: "a6ca6242d6744e37a9d2b1aa76faef3c"
format_stamp: "Formatted at 2026-08-06 00:58:21 on dist-test-slave-dhph"
server_key: "60480817314b3f3b238f4f1829597812"
server_key_iv: "e4d214bd06e8b6584605bfd2f8393de4"
server_key_version: "encryptionkey@0"
I20260806 00:58:22.132725 31296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:22.141362 31296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:22.143095 31296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:22.146751 31296 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.61:41127
I20260806 00:58:22.147069 31467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.61:41127 every 8 connection(s)
I20260806 00:58:22.147238 31296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-1/data/info.pb
I20260806 00:58:22.148176 31468 sys_catalog.cc:263] Verifying existing consensus state
I20260806 00:58:22.148694 31468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c: Bootstrap starting.
I20260806 00:58:22.151829 31468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 00:58:22.152086 31468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c: Bootstrap complete.
I20260806 00:58:22.152354 31468 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: NON_VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: true } }
I20260806 00:58:22.152446 31468 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a6ca6242d6744e37a9d2b1aa76faef3c, State: Initialized, Role: LEARNER
I20260806 00:58:22.152482 31468 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: NON_VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: true } }
I20260806 00:58:22.152642 31468 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 00:58:22.154026 31472 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: NON_VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: true } } }
I20260806 00:58:22.154121 31472 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [sys.catalog]: This master's current role is: LEARNER
I20260806 00:58:22.154225 31484 catalog_manager.cc:1259] Loaded cluster ID: 3b0f30832a7d4834950ee317a3e241e2
I20260806 00:58:22.154277 31484 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c: loading cluster ID for follower catalog manager: success
I20260806 00:58:22.155653 31484 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c: acquiring CA information for follower catalog manager: success
I20260806 00:58:22.155956 31484 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 00:58:22.161464 31471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.152246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33878 (local address 127.23.242.61:41127)
0806 00:58:22.152391 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.152393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.152400 (+ 7us) server_negotiation.cc:407] Connection header received
0806 00:58:22.152423 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.152424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.152437 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.153159 (+ 722us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.153206 (+ 47us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.154122 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.155097 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.155408 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.155479 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.157457 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.157460 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.157460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.157472 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:22.159676 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.160241 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.160244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.160244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.160355 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.160657 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.160658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.160658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.160799 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:22.160986 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:22.161101 (+ 115us) server_negotiation.cc:299] Negotiation successful
0806 00:58:22.161346 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for oryx/127.23.242.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.242.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.242.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.60.keytab.
Entry for principal oryx/127.23.242.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.60.keytab.
I20260806 00:58:22.193699 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.60:41799
--webserver_interface=127.23.242.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.60.keytab
--principal=oryx/127.23.242.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:38515,127.23.242.61:41127,127.23.242.60:41799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:22.303907 31487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:22.304119 31487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:22.304150 31487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:22.305389 31487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 00:58:22.305437 31487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:22.305451 31487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:22.305465 31487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 00:58:22.305477 31487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 00:58:22.307344 31487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:38515,127.23.242.61:41127,127.23.242.60:41799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.60.keytab
--principal=oryx/127.23.242.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.60:41799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.242.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:22.307596 31487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:22.307801 31487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:22.313313 31493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:22.313310 31494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:22.313340 31496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:22.313870 31487 server_base.cc:1034] running on GCE node
I20260806 00:58:22.314174 31487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:22.314486 31487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:22.315722 31487 hybrid_clock.cc:648] HybridClock initialized: now 1785977902315698 us; error 42 us; skew 500 ppm
Aug 06 00:58:22 dist-test-slave-dhph krb5kdc[30777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977902, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:22.318867 31487 init.cc:377] Logged in from keytab as oryx/127.23.242.60@KRBTEST.COM (short username oryx)
I20260806 00:58:22.319325 31487 webserver.cc:460] Webserver started at http://127.23.242.60:43389/ using document root <none> and password file <none>
I20260806 00:58:22.319538 31487 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:22.319579 31487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:22.319782 31487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:22.321053 31487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-2/data/instance:
uuid: "12658ab315ca4652baab3a5f69d785e3"
format_stamp: "Formatted at 2026-08-06 00:58:22 on dist-test-slave-dhph"
server_key: "65fb2eb38ec8da4ced2d5bf517fffe02"
server_key_iv: "58bfe56a8a74b996d2f5f19e8451d51a"
server_key_version: "encryptionkey@0"
I20260806 00:58:22.321516 31487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-2/wal/instance:
uuid: "12658ab315ca4652baab3a5f69d785e3"
format_stamp: "Formatted at 2026-08-06 00:58:22 on dist-test-slave-dhph"
server_key: "65fb2eb38ec8da4ced2d5bf517fffe02"
server_key_iv: "58bfe56a8a74b996d2f5f19e8451d51a"
server_key_version: "encryptionkey@0"
I20260806 00:58:22.323339 31487 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260806 00:58:22.324472 31503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:22.324913 31487 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:22.325037 31487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-2/wal
uuid: "12658ab315ca4652baab3a5f69d785e3"
format_stamp: "Formatted at 2026-08-06 00:58:22 on dist-test-slave-dhph"
server_key: "65fb2eb38ec8da4ced2d5bf517fffe02"
server_key_iv: "58bfe56a8a74b996d2f5f19e8451d51a"
server_key_version: "encryptionkey@0"
I20260806 00:58:22.325127 31487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:22.343312 31487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:22.346765 31487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:22.346979 31487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:22.352548 31487 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.60:41799
I20260806 00:58:22.352640 31555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.60:41799 every 8 connection(s)
I20260806 00:58:22.353201 31487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-2/data/info.pb
I20260806 00:58:22.354669 31556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:22.356477 31556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } has no permanent_uuid. Determining permanent_uuid...
I20260806 00:58:22.361824 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 31487
Aug 06 00:58:22 dist-test-slave-dhph krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977902, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.60@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM
Aug 06 00:58:22 dist-test-slave-dhph krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.242.60@KRBTEST.COM
I20260806 00:58:22.370337 30865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.357364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57546 (local address 127.23.242.62:38515)
0806 00:58:22.357468 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.357470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.357628 (+ 158us) server_negotiation.cc:407] Connection header received
0806 00:58:22.358260 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.358261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.358275 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.358341 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.359317 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.360411 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.362185 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.362447 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.362536 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.366349 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.366351 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.366351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.366360 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:22.368475 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.369027 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.369029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.369029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.369103 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.369493 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.369495 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.369495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.369651 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:22.369736 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:22.370043 (+ 307us) server_negotiation.cc:299] Negotiation successful
0806 00:58:22.370220 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 00:58:22.370623 31558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.357202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.0.0.1:57546)
0806 00:58:22.357459 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:22.357515 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:22.358106 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:22.358423 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:22.358430 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:22.358827 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:22.359181 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.359188 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.361026 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.361028 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.362068 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.362071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.362762 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.362763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.362938 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.363750 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:22.363758 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:22.366191 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:22.368600 (+ 2409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.368611 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.368614 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.368935 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.369241 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.369242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.369243 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.369401 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.369725 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:22.369733 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:22.369889 (+ 156us) client_negotiation.cc:769] Sending connection context
0806 00:58:22.370008 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 00:58:22.370302 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":121,"threads_started":1}
I20260806 00:58:22.371522 31556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } has no permanent_uuid. Determining permanent_uuid...
I20260806 00:58:22.373603 31559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.362425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49488 (local address 127.23.242.60:41799)
0806 00:58:22.362785 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.362790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.362817 (+ 27us) server_negotiation.cc:407] Connection header received
0806 00:58:22.362841 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.362843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.362879 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.362987 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.363939 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.364928 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.366206 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.366688 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.366756 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.369744 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.369753 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.369753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.369761 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:22.372092 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.372492 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.372495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.372496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.372558 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.372798 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.372799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.372799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.372947 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:22.373074 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:22.373381 (+ 307us) server_negotiation.cc:299] Negotiation successful
0806 00:58:22.373498 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":131,"threads_started":1}
Aug 06 00:58:22 dist-test-slave-dhph krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977902, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.60@KRBTEST.COM for oryx/127.23.242.61@KRBTEST.COM
I20260806 00:58:22.381062 31558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.371889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:41127 (local address 127.0.0.1:33882)
0806 00:58:22.372021 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:22.372033 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:22.372095 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:22.372263 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:22.372264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:22.372370 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:22.372423 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.372424 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.373469 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.373470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.374214 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.374217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.374632 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.374633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.374709 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.375171 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:22.375175 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:22.376751 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:22.379365 (+ 2614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.379366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.379366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.379546 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.380046 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.380047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.380047 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.380171 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.380576 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:22.380577 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:22.380808 (+ 231us) client_negotiation.cc:769] Sending connection context
0806 00:58:22.380891 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 00:58:22.380987 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260806 00:58:22.381140 31471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.371967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33882 (local address 127.23.242.61:41127)
0806 00:58:22.372015 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.372017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.372074 (+ 57us) server_negotiation.cc:407] Connection header received
0806 00:58:22.372182 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.372183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.372196 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.372250 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.372521 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.373376 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.374282 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.374511 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.374614 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.376851 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.376852 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.376852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.376859 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:22.379259 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.379745 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.379747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.379748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.379900 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.380346 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.380348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.380348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.380488 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:22.380560 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:22.380963 (+ 403us) server_negotiation.cc:299] Negotiation successful
0806 00:58:22.381054 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 00:58:22.381784 31556 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.60" port: 41799 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 00:58:22 dist-test-slave-dhph krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977902, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.60@KRBTEST.COM for oryx/127.23.242.60@KRBTEST.COM
I20260806 00:58:22.389621 31559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.382184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49492 (local address 127.23.242.60:41799)
0806 00:58:22.382346 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.382349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.382429 (+ 80us) server_negotiation.cc:407] Connection header received
0806 00:58:22.382510 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.382511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.382526 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.382700 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.382986 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.383513 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.384238 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.384458 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.384536 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.386939 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.386941 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.386941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.386950 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:22.388390 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.388786 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.388787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.388788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.388843 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.389042 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.389043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.389044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.389162 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:22.389234 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:22.389383 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 00:58:22.389499 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":141}
I20260806 00:58:22.389640 31558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.382121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:41799 (local address 127.0.0.1:49492)
0806 00:58:22.382352 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:22.382365 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:22.382447 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:22.382707 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:22.382708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:22.382867 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:22.382903 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.382904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.383622 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.383623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.384137 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.384138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.384544 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.384545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.384599 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.385149 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:22.385156 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:22.386826 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:22.388507 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.388508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.388508 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.388698 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.388909 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.388910 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.388910 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.388969 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.389236 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:22.389237 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:22.389275 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 00:58:22.389360 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 00:58:22.389550 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":96}
I20260806 00:58:22.391516 31556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3: Bootstrap starting.
I20260806 00:58:22.392589 31556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:22.392933 31556 log.cc:824] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:22.393986 31556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3: No bootstrap required, opened a new log
I20260806 00:58:22.395676 31556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 41799 } }
I20260806 00:58:22.395836 31556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:22.395889 31556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12658ab315ca4652baab3a5f69d785e3, State: Initialized, Role: FOLLOWER
I20260806 00:58:22.396062 31556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 41799 } }
I20260806 00:58:22.396428 31556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 00:58:22.396452 31561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 41799 } } }
I20260806 00:58:22.396606 31561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 00:58:22.398401 31573 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 00:58:22.398654 31573 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 00:58:22.402428 31487 master_runner.cc:428] Detected that this master 12658ab315ca4652baab3a5f69d785e3 is joining an existing cluster
I20260806 00:58:22.402503 31487 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.242.60:41799
I20260806 00:58:22.412155 30865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.403718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57554 (local address 127.23.242.62:38515)
0806 00:58:22.403780 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.403782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.403942 (+ 160us) server_negotiation.cc:407] Connection header received
0806 00:58:22.404039 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.404040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.404055 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.404095 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.404732 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.405591 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.407256 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.407609 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.407656 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.409240 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.409250 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.409250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.409258 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:22.410808 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.411203 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.411204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.411204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.411280 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.411527 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.411528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.411529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.411680 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:22.411771 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:22.411956 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 00:58:22.412041 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260806 00:58:22.412174 31579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.403660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.0.0.1:57554)
0806 00:58:22.403848 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:22.403859 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:22.403959 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:22.404195 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:22.404196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:22.404504 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:22.404615 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.404617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.405698 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.405699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.407128 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.407131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.407790 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.407791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.407865 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.408523 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:22.408530 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:22.409112 (+ 582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:22.410916 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.410918 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.410918 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.411118 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.411363 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.411364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.411365 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.411452 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.411762 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:22.411763 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:22.411816 (+ 53us) client_negotiation.cc:769] Sending connection context
0806 00:58:22.411913 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 00:58:22.412043 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":114,"thread_start_us":120,"threads_started":1}
Aug 06 00:58:22 dist-test-slave-dhph krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977901, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.62@KRBTEST.COM for oryx/127.23.242.60@KRBTEST.COM
I20260806 00:58:22.425467 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 30887
I20260806 00:58:22.432390 31559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.404134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49494 (local address 127.23.242.60:41799)
0806 00:58:22.404341 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.404343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.404448 (+ 105us) server_negotiation.cc:407] Connection header received
0806 00:58:22.404528 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.404529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.404536 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.404562 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.404983 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.405696 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.407397 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.407719 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.407769 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.428219 (+ 20450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.428221 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.428221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.428229 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:22.430515 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.430925 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.430926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.430926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.430977 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.431650 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.431651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.431651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.431819 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:22.431879 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:22.432091 (+ 212us) server_negotiation.cc:299] Negotiation successful
0806 00:58:22.432234 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260806 00:58:22.433452 31580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.403921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:41799 (local address 127.0.0.1:49494)
0806 00:58:22.404393 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:22.404408 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:22.404468 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:22.404611 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:22.404611 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:22.404834 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:22.404897 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.404899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.405786 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.405787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.407279 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.407282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.407850 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.407850 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.407914 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.427533 (+ 19619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:22.427540 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:22.428040 (+ 500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:22.430630 (+ 2590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.430631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.430631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.430845 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.431062 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.431062 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.431063 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.431556 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.431910 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:22.431911 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:22.431951 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 00:58:22.433179 (+ 1228us) client_negotiation.cc:240] Negotiation successful
0806 00:58:22.433327 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":258,"thread_start_us":63,"threads_started":1}
I20260806 00:58:22.434091 31471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.404101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33898 (local address 127.23.242.61:41127)
0806 00:58:22.404173 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.404175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.404186 (+ 11us) server_negotiation.cc:407] Connection header received
0806 00:58:22.404227 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.404228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.404238 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.404294 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.404724 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.405583 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.407255 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.407608 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.407656 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.429645 (+ 21989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.429650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.429650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.429667 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:22.432535 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.433083 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.433084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.433085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.433149 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.433393 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.433394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.433394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.433554 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:22.433602 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:22.433848 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 00:58:22.433942 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 00:58:22.434556 31581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.403715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:41127 (local address 127.0.0.1:33898)
0806 00:58:22.404057 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:22.404068 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:22.404155 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:22.404308 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:22.404309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:22.404540 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:22.404615 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.404617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.405698 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.405699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.407128 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.407131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.408126 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.408128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.408203 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.428286 (+ 20083us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:22.428294 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:22.428933 (+ 639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:22.432660 (+ 3727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.432661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.432673 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.432970 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.433241 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.433242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.433242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.433313 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.433651 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:22.433652 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:22.433710 (+ 58us) client_negotiation.cc:769] Sending connection context
0806 00:58:22.434240 (+ 530us) client_negotiation.cc:240] Negotiation successful
0806 00:58:22.434374 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":276,"thread_start_us":87,"threads_started":1}
I20260806 00:58:22.438402 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.414769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:41799 (local address 127.0.0.1:49498)
0806 00:58:22.414828 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:22.414839 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:22.414877 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:22.415179 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:22.415180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:22.415374 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:22.415430 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.415432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.416241 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.416242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.417178 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.417181 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.417619 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.417620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.417704 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.418260 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:22.418266 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:22.419862 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:22.437134 (+ 17272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.437136 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.437137 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.437423 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.437667 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.437667 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.437668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.437731 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.438038 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:22.438039 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:22.438094 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 00:58:22.438147 (+ 53us) client_negotiation.cc:240] Negotiation successful
0806 00:58:22.438287 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260806 00:58:22.438719 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:42435
--local_ip_for_outbound_sockets=127.23.242.1
--tserver_master_addrs=127.23.242.62:38515
--webserver_port=33405
--webserver_interface=127.23.242.1
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.1.keytab
--principal=oryx/127.23.242.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.242.62:38515,127.23.242.61:41127
--tserver_master_addrs=127.23.242.62:38515,127.23.242.61:41127,127.23.242.60:41799 with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 00:58:22.439234 31582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.414853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49498 (local address 127.23.242.60:41799)
0806 00:58:22.415044 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.415046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.415052 (+ 6us) server_negotiation.cc:407] Connection header received
0806 00:58:22.415081 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.415082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.415091 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.415145 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.415546 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.416150 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.417277 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.417528 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.417605 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.419997 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.419999 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.419999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.420006 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:22.436994 (+ 16988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.437524 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.437525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.437525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.437586 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.437807 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.437808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.437808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.437954 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:22.438937 (+ 983us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:22.438984 (+ 47us) server_negotiation.cc:299] Negotiation successful
0806 00:58:22.439079 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":137,"threads_started":1}
I20260806 00:58:22.439805 30822 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.242.60:41799: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "12658ab315ca4652baab3a5f69d785e3"
member_type: NON_VOTER
last_known_addr {
host: "127.23.242.60"
port: 41799
}
attrs {
promote: true
}
}
dest_uuid: "ee13c8d77d44483c8dec48001f37be48"
cas_config_opid_index: 5
I20260806 00:58:22.440100 30822 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: NON_VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: true } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: NON_VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: true } }
I20260806 00:58:22.440644 31393 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 12658ab315ca4652baab3a5f69d785e3 (127.23.242.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: NON_VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: true } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: NON_VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: true } } }
I20260806 00:58:22.442054 31393 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.242.60:41799
I20260806 00:58:22.442142 31393 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ee13c8d77d44483c8dec48001f37be48" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: NON_VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: true } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: NON_VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: true } } }
I20260806 00:58:22.442255 31393 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:22.442308 31530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:22.442446 31487 master.cc:537] Master@127.23.242.60:41799 shutting down...
I20260806 00:58:22.443742 31530 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 1 FOLLOWER]: Refusing update from remote peer ee13c8d77d44483c8dec48001f37be48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 00:58:22.444150 31392 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: NON_VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 00:58:22.445179 30808 consensus_peers.cc:487] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 -> Peer 12658ab315ca4652baab3a5f69d785e3 (127.23.242.60:41799): Couldn't send request to peer 12658ab315ca4652baab3a5f69d785e3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 00:58:22.448089 31487 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:22.448208 31487 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:22.448266 31487 tablet_replica.cc:330] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3: stopping tablet replica
I20260806 00:58:22.451506 31471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.444057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33906 (local address 127.23.242.61:41127)
0806 00:58:22.444129 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.444131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.445435 (+ 1304us) server_negotiation.cc:407] Connection header received
0806 00:58:22.445567 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.445569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.445739 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.445814 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:22.446765 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.447680 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.449903 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.450834 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.450901 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.450984 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:22.451350 (+ 366us) server_negotiation.cc:299] Negotiation successful
0806 00:58:22.451403 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260806 00:58:22.451871 31487 master.cc:552] Master@127.23.242.60:41799 shutdown complete.
I20260806 00:58:22.451962 31487 master_runner.cc:305] Clearing existing system tablet
I20260806 00:58:22.452075 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.442070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:41127 (local address 127.0.0.1:33906)
0806 00:58:22.445351 (+ 3281us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:22.445368 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:22.445471 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:22.446401 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:22.446403 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:22.446405 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:22.446467 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.446471 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.447820 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.447821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.449768 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.449773 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.451137 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.451138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.451222 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.451251 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 00:58:22.451839 (+ 588us) client_negotiation.cc:240] Negotiation successful
0806 00:58:22.451901 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3150}
I20260806 00:58:22.452117 31487 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:22.452165 31487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:22.452569 31442 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 1 LEARNER]: Refusing update from remote peer ee13c8d77d44483c8dec48001f37be48: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 00:58:22.452919 31392 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: NON_VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 00:58:22.453219 31487 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:22.453635 31442 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 12658ab315ca4652baab3a5f69d785e3 (127.23.242.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: NON_VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: true } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: NON_VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: true } } }
I20260806 00:58:22.455060 31587 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee13c8d77d44483c8dec48001f37be48" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: NON_VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: true } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: NON_VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: true } } }
I20260806 00:58:22.455159 31587 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [sys.catalog]: This master's current role is: LEARNER
I20260806 00:58:22.455325 31472 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee13c8d77d44483c8dec48001f37be48. Latest consensus state: current_term: 1 leader_uuid: "ee13c8d77d44483c8dec48001f37be48" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: NON_VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: true } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: NON_VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: true } } }
I20260806 00:58:22.455425 31472 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [sys.catalog]: This master's current role is: LEARNER
I20260806 00:58:22.455811 31392 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48: attempting to promote NON_VOTER a6ca6242d6744e37a9d2b1aa76faef3c to VOTER
I20260806 00:58:22.456001 31588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:22.456048 31392 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: NON_VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: true } }
I20260806 00:58:22.456135 31487 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:22.456175 31487 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260806 00:58:22.456197 31487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-2/wal
uuid: "12658ab315ca4652baab3a5f69d785e3"
format_stamp: "Formatted at 2026-08-06 00:58:22 on dist-test-slave-dhph"
server_key: "65fb2eb38ec8da4ced2d5bf517fffe02"
server_key_iv: "58bfe56a8a74b996d2f5f19e8451d51a"
server_key_version: "encryptionkey@0"
I20260806 00:58:22.456722 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.456567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:41799 (local address 127.0.0.1:49512)
0806 00:58:22.456644 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:22.456689 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:41799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 00:58:22.456902 31442 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 1 LEARNER]: Refusing update from remote peer ee13c8d77d44483c8dec48001f37be48: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260806 00:58:22.456969 31487 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:22.457185 31584 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 00:58:22.457700 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.457375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:41799 (local address 127.0.0.1:49522)
0806 00:58:22.457639 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:22.457670 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:41799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247}
W20260806 00:58:22.457782 30808 consensus_peers.cc:487] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 -> Peer 12658ab315ca4652baab3a5f69d785e3 (127.23.242.60:41799): Couldn't send request to peer 12658ab315ca4652baab3a5f69d785e3. Status: Network error: Client connection negotiation failed: client connection to 127.23.242.60:41799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 00:58:22.458110 31584 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a6ca6242d6744e37a9d2b1aa76faef3c (127.23.242.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: NON_VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: true } } }
I20260806 00:58:22.458724 31487 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 00:58:22.458791 31487 log.cc:1196] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 00:58:22.458992 31487 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3: Deleting consensus metadata
I20260806 00:58:22.459107 31487 master_runner.cc:315] Copying system tablet from 127.23.242.62:38515
I20260806 00:58:22.459156 31584 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ee13c8d77d44483c8dec48001f37be48" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: NON_VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: true } } }
I20260806 00:58:22.459234 31584 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:22.459290 31584 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee13c8d77d44483c8dec48001f37be48" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: NON_VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: true } } }
I20260806 00:58:22.459334 31584 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:22.459640 31442 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a6ca6242d6744e37a9d2b1aa76faef3c (127.23.242.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: NON_VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: true } } }
I20260806 00:58:22.459725 31487 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3: tablet copy: Beginning tablet copy session from remote peer at address 127.23.242.62:38515
I20260806 00:58:22.460626 31472 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee13c8d77d44483c8dec48001f37be48" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: NON_VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: true } } }
I20260806 00:58:22.460713 31472 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 00:58:22.468780 31594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.460395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.0.0.1:57568)
0806 00:58:22.460560 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:22.460571 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:22.460643 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:22.460908 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:22.460909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:22.461129 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:22.461398 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.461402 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.462447 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.462449 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.463995 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.463998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.464507 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.464509 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.464587 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.465127 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:22.465133 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:22.465554 (+ 421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:22.467506 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.467508 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.467508 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.467737 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.467955 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.467956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.467956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.468045 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.468383 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:22.468385 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:22.468434 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 00:58:22.468507 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 00:58:22.468631 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":38,"threads_started":1}
I20260806 00:58:22.469256 30865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.460489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57568 (local address 127.23.242.62:38515)
0806 00:58:22.460534 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.460536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.460619 (+ 83us) server_negotiation.cc:407] Connection header received
0806 00:58:22.460724 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.460725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.460737 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.460767 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.461504 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.462333 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.464097 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.464406 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.464490 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.465658 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.465659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.465660 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.465668 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:22.467420 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.467816 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.467817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.467818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.467882 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.468134 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.468135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.468135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.468280 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:22.468351 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:22.468548 (+ 197us) server_negotiation.cc:299] Negotiation successful
0806 00:58:22.468739 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 00:58:22.469646 30844 tablet_copy_service.cc:140] P ee13c8d77d44483c8dec48001f37be48: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 12658ab315ca4652baab3a5f69d785e3 ({username='oryx', principal='oryx/127.23.242.60@KRBTEST.COM'} at 127.0.0.1:57568)
I20260806 00:58:22.469719 30844 tablet_copy_service.cc:161] P ee13c8d77d44483c8dec48001f37be48: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 12658ab315ca4652baab3a5f69d785e3 at {username='oryx', principal='oryx/127.23.242.60@KRBTEST.COM'} at 127.0.0.1:57568: session id = 12658ab315ca4652baab3a5f69d785e3-00000000000000000000000000000000
I20260806 00:58:22.470367 30844 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 00:58:22.470958 31487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 00:58:22.472365 31487 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3: tablet copy: Starting download of 0 data blocks...
I20260806 00:58:22.472537 31487 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3: tablet copy: Starting download of 1 WAL segments...
I20260806 00:58:22.473433 31487 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 00:58:22.474424 30844 tablet_copy_service.cc:342] P ee13c8d77d44483c8dec48001f37be48: Request end of tablet copy session 12658ab315ca4652baab3a5f69d785e3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.242.60@KRBTEST.COM'} at 127.0.0.1:57568
I20260806 00:58:22.474484 30844 tablet_copy_service.cc:433] P ee13c8d77d44483c8dec48001f37be48: ending tablet copy session 12658ab315ca4652baab3a5f69d785e3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 12658ab315ca4652baab3a5f69d785e3
I20260806 00:58:22.478406 31487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:22.480000 31595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:22.483780 31596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:22.483966 31598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:22.486766 31487 server_base.cc:1034] running on GCE node
I20260806 00:58:22.486887 31487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:22.487043 31487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:22.488181 31487 hybrid_clock.cc:648] HybridClock initialized: now 1785977902488163 us; error 46 us; skew 500 ppm
Aug 06 00:58:22 dist-test-slave-dhph krb5kdc[30777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977902, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:22.489866 31487 init.cc:377] Logged in from keytab as oryx/127.23.242.60@KRBTEST.COM (short username oryx)
I20260806 00:58:22.490234 31487 webserver.cc:460] Webserver started at http://127.23.242.60:45087/ using document root <none> and password file <none>
I20260806 00:58:22.490363 31487 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:22.490406 31487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:22.491271 31487 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260806 00:58:22.491905 31605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:22.492020 31487 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 00:58:22.492056 31487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-2/wal
uuid: "12658ab315ca4652baab3a5f69d785e3"
format_stamp: "Formatted at 2026-08-06 00:58:22 on dist-test-slave-dhph"
server_key: "65fb2eb38ec8da4ced2d5bf517fffe02"
server_key_iv: "58bfe56a8a74b996d2f5f19e8451d51a"
server_key_version: "encryptionkey@0"
I20260806 00:58:22.492116 31487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:22.505647 31487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:22.509656 31487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:22.513595 31487 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.60:41799
I20260806 00:58:22.514099 31487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/master-2/data/info.pb
I20260806 00:58:22.514256 31659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.60:41799 every 8 connection(s)
I20260806 00:58:22.515278 31660 sys_catalog.cc:263] Verifying existing consensus state
I20260806 00:58:22.515799 31660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3: Bootstrap starting.
I20260806 00:58:22.518839 31660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 00:58:22.519073 31660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3: Bootstrap complete.
I20260806 00:58:22.519328 31660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: NON_VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: true } }
I20260806 00:58:22.519426 31660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 12658ab315ca4652baab3a5f69d785e3, State: Initialized, Role: LEARNER
I20260806 00:58:22.519466 31660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: NON_VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: true } }
I20260806 00:58:22.519678 31660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 00:58:22.519714 31663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: NON_VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: true } } }
I20260806 00:58:22.519989 31663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [sys.catalog]: This master's current role is: LEARNER
I20260806 00:58:22.522857 31674 catalog_manager.cc:1259] Loaded cluster ID: 3b0f30832a7d4834950ee317a3e241e2
I20260806 00:58:22.523182 31674 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3: loading cluster ID for follower catalog manager: success
I20260806 00:58:22.524621 31674 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3: acquiring CA information for follower catalog manager: success
I20260806 00:58:22.525007 31674 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 00:58:22.563409 31583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:22.563674 31583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:22.563726 31583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:22.564980 31583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:22.565055 31583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:22.565106 31583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 00:58:22.566932 31583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.1.keytab
--principal=oryx/127.23.242.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:42435
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=33405
--tserver_master_addrs=127.23.242.62:38515,127.23.242.61:41127,127.23.242.60:41799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:22.567209 31583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:22.567461 31583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:22.574055 31680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:22.574229 31681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:22.574038 31683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:22.574887 31583 server_base.cc:1034] running on GCE node
I20260806 00:58:22.575158 31583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:22.575467 31583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:22.576660 31583 hybrid_clock.cc:648] HybridClock initialized: now 1785977902576640 us; error 34 us; skew 500 ppm
Aug 06 00:58:22 dist-test-slave-dhph krb5kdc[30777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977902, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:22.580601 31583 init.cc:377] Logged in from keytab as oryx/127.23.242.1@KRBTEST.COM (short username oryx)
I20260806 00:58:22.581173 31583 webserver.cc:460] Webserver started at http://127.23.242.1:33405/ using document root <none> and password file <none>
I20260806 00:58:22.581442 31583 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:22.581549 31583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:22.583618 31583 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 00:58:22.584720 31690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:22.585002 31583 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:22.585187 31583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "ba9ec1e1be664d3b9e705abaf8f50594"
format_stamp: "Formatted at 2026-08-06 00:58:21 on dist-test-slave-dhph"
server_key: "bf11ceecb7960f30dfa81fd30af85064"
server_key_iv: "1bfb90eff4e80ce3a0528a0fd97d9b9a"
server_key_version: "encryptionkey@0"
I20260806 00:58:22.585671 31583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:22.600541 31583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:22.604635 31583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:22.604877 31583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:22.605168 31583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:22.605566 31583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:22.605625 31583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:22.605665 31583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:22.605692 31583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:22.616973 31583 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:42435
I20260806 00:58:22.617115 31807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:42435 every 8 connection(s)
I20260806 00:58:22.617669 31583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 00:58:22.617851 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 31583
I20260806 00:58:22.618052 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 31024
Aug 06 00:58:22 dist-test-slave-dhph krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977902, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.61@KRBTEST.COM
I20260806 00:58:22.630570 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:45035
--local_ip_for_outbound_sockets=127.23.242.2
--tserver_master_addrs=127.23.242.62:38515
--webserver_port=35943
--webserver_interface=127.23.242.2
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.2.keytab
--principal=oryx/127.23.242.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.242.62:38515,127.23.242.61:41127
--tserver_master_addrs=127.23.242.62:38515,127.23.242.61:41127,127.23.242.60:41799 with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 00:58:22 dist-test-slave-dhph krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977902, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.60@KRBTEST.COM
Aug 06 00:58:22 dist-test-slave-dhph krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977902, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM
I20260806 00:58:22.635613 31812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.620883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:41127 (local address 127.23.242.1:48411)
0806 00:58:22.621104 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:22.621176 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:22.622201 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:22.622662 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:22.622667 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:22.623759 (+ 1092us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:22.624170 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.624178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.625018 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.625019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.626310 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.626313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.626958 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.626960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.627139 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.627902 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:22.627913 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:22.630454 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:22.633346 (+ 2892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.633357 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.633360 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.633947 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.634220 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.634221 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.634221 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.634375 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.634804 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:22.634812 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:22.634993 (+ 181us) client_negotiation.cc:769] Sending connection context
0806 00:58:22.635106 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 00:58:22.635309 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":94,"threads_started":1}
I20260806 00:58:22.635718 31814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.621388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:40707 (local address 127.23.242.60:41799)
0806 00:58:22.621525 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.621527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.624839 (+ 3312us) server_negotiation.cc:407] Connection header received
0806 00:58:22.625021 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.625023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.625042 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.625126 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.625543 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.626814 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.628184 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.628592 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.628647 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.631309 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.631310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.631311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.631319 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:22.633988 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.634463 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.634464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.634465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.634534 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.634843 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.634845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.634845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.635013 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:22.635062 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:22.635302 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 00:58:22.635406 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
I20260806 00:58:22.636157 31471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.620850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:48411 (local address 127.23.242.61:41127)
0806 00:58:22.620929 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.620932 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.621293 (+ 361us) server_negotiation.cc:407] Connection header received
0806 00:58:22.622436 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.622442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.622457 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.622510 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.624293 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.624925 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.626443 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.626804 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.626878 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.630611 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.630613 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.630614 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.630624 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:22.633233 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.634072 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.634073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.634073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.634145 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.634460 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.634461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.634462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.634702 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:22.634772 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:22.635924 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0806 00:58:22.636022 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 00:58:22.636251 31815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.621320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:41799 (local address 127.23.242.1:40707)
0806 00:58:22.624760 (+ 3440us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:22.624772 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:22.624866 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:22.625164 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:22.625165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:22.625376 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:22.625447 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.625449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.626929 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.626931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.628069 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.628073 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.628751 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.628752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.628832 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.629361 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:22.629367 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:22.631186 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:22.634098 (+ 2912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.634099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.634100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.634351 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.634675 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.634676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.634677 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.634753 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.635117 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:22.635118 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:22.635167 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 00:58:22.635999 (+ 832us) client_negotiation.cc:240] Negotiation successful
0806 00:58:22.636140 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3417,"thread_start_us":97,"threads_started":1}
I20260806 00:58:22.636670 31808 heartbeater.cc:347] Connected to a master server at 127.23.242.60:41799
I20260806 00:58:22.636682 31810 heartbeater.cc:347] Connected to a master server at 127.23.242.61:41127
I20260806 00:58:22.636816 31810 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:22.636817 31808 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:22.637079 31810 heartbeater.cc:511] Master 127.23.242.61:41127 requested a full tablet report, sending...
I20260806 00:58:22.637125 31808 heartbeater.cc:511] Master 127.23.242.60:41799 requested a full tablet report, sending...
I20260806 00:58:22.637737 31623 ts_manager.cc:194] Registered new tserver with Master: ba9ec1e1be664d3b9e705abaf8f50594 (127.23.242.1:42435)
I20260806 00:58:22.638149 31432 ts_manager.cc:194] Registered new tserver with Master: ba9ec1e1be664d3b9e705abaf8f50594 (127.23.242.1:42435)
I20260806 00:58:22.639129 31813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.624562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.23.242.1:33965)
0806 00:58:22.627313 (+ 2751us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:22.627328 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:22.627434 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:22.627646 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:22.627647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:22.627886 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:22.627948 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.627951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.629056 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.629057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.630023 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.630027 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.630691 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.630692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.630756 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.631286 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:22.631292 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:22.633340 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:22.636744 (+ 3404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.636745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.636745 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.637506 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.638141 (+ 635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.638142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.638142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.638249 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.638744 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:22.638745 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:22.638795 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 00:58:22.638882 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 00:58:22.639002 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2727,"thread_start_us":124,"threads_started":1}
I20260806 00:58:22.639174 30865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.624799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:33965 (local address 127.23.242.62:38515)
0806 00:58:22.624853 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.624856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.627396 (+ 2540us) server_negotiation.cc:407] Connection header received
0806 00:58:22.627523 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.627524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.627539 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.627572 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.628062 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.628881 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.630137 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.630508 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.630558 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.633473 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.633475 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.633476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.633485 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:22.635507 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.637675 (+ 2168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.637678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.637678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.637798 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.638419 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.638421 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.638422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.638664 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:22.638757 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:22.638937 (+ 180us) server_negotiation.cc:299] Negotiation successful
0806 00:58:22.639043 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 00:58:22.639780 31809 heartbeater.cc:347] Connected to a master server at 127.23.242.62:38515
I20260806 00:58:22.639858 31809 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:22.639950 31809 heartbeater.cc:511] Master 127.23.242.62:38515 requested a full tablet report, sending...
I20260806 00:58:22.640638 30822 ts_manager.cc:194] Re-registered known tserver with Master: ba9ec1e1be664d3b9e705abaf8f50594 (127.23.242.1:42435)
I20260806 00:58:22.641251 30822 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.242.1@KRBTEST.COM'} at 127.23.242.1:33965
W20260806 00:58:22.734442 31816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:22.734807 31816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:22.734844 31816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:22.737183 31816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:22.737299 31816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:22.737346 31816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 00:58:22.739681 31816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.2.keytab
--principal=oryx/127.23.242.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:45035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=35943
--tserver_master_addrs=127.23.242.62:38515,127.23.242.61:41127,127.23.242.60:41799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:22.740020 31816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:22.740345 31816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:22.746587 31823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:22.746702 31822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:22.746815 31826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:22.746845 31816 server_base.cc:1034] running on GCE node
I20260806 00:58:22.747478 31816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:22.747898 31816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:22.749186 31816 hybrid_clock.cc:648] HybridClock initialized: now 1785977902749118 us; error 57 us; skew 500 ppm
Aug 06 00:58:22 dist-test-slave-dhph krb5kdc[30777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977902, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:22.752982 31816 init.cc:377] Logged in from keytab as oryx/127.23.242.2@KRBTEST.COM (short username oryx)
I20260806 00:58:22.754489 31816 webserver.cc:460] Webserver started at http://127.23.242.2:35943/ using document root <none> and password file <none>
I20260806 00:58:22.755056 31816 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:22.755173 31816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:22.756839 31816 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 00:58:22.757786 31833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:22.758101 31816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:22.758234 31816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "4765fb0b489240bb920242b63a57e71f"
format_stamp: "Formatted at 2026-08-06 00:58:21 on dist-test-slave-dhph"
server_key: "234cd73e37450275ff372942ddfca717"
server_key_iv: "6c71b11e8cdda8b72761829a2c0dd594"
server_key_version: "encryptionkey@0"
I20260806 00:58:22.758672 31816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:22.781224 31816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:22.783811 31816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:22.783939 31816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:22.784142 31816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:22.784490 31816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:22.784550 31816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:22.784588 31816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:22.784643 31816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:22.791666 31816 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:45035
I20260806 00:58:22.791704 31947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:45035 every 8 connection(s)
I20260806 00:58:22.792130 31816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 00:58:22.799711 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 31816
Aug 06 00:58:22 dist-test-slave-dhph krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977902, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM
Aug 06 00:58:22 dist-test-slave-dhph krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977902, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.61@KRBTEST.COM
I20260806 00:58:22.801648 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 31160
Aug 06 00:58:22 dist-test-slave-dhph krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977902, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.60@KRBTEST.COM
I20260806 00:58:22.806335 30865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.793020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:44381 (local address 127.23.242.62:38515)
0806 00:58:22.793070 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.793073 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.793469 (+ 396us) server_negotiation.cc:407] Connection header received
0806 00:58:22.794419 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.794421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.794439 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.794524 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.795503 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.796072 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.797739 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.798088 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.798162 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.802378 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.802381 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.802381 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.802389 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:22.804245 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.804810 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.804811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.804812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.804874 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.805253 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.805254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.805254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.805412 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:22.805461 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:22.805972 (+ 511us) server_negotiation.cc:299] Negotiation successful
0806 00:58:22.806072 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 00:58:22.806335 31471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.792954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:34691 (local address 127.23.242.61:41127)
0806 00:58:22.793037 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.793041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.793300 (+ 259us) server_negotiation.cc:407] Connection header received
0806 00:58:22.794233 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.794236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.794262 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.794359 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.796231 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.797029 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.798154 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.798485 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.798562 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.801994 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.801995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.801996 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.802005 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:22.804055 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.804807 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.804808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.804808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.804873 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.805253 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.805254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.805254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.805412 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:22.805461 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:22.805970 (+ 509us) server_negotiation.cc:299] Negotiation successful
0806 00:58:22.806072 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 00:58:22.807236 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.792960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:41127 (local address 127.23.242.2:34691)
0806 00:58:22.793159 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:22.793211 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:22.793933 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:22.794496 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:22.794501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:22.795225 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:22.795384 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.795390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.797197 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.797199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.798045 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.798048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.798603 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.798604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.798686 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.799652 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:22.799656 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:22.801819 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:22.804175 (+ 2356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.804186 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.804189 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.804699 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.804964 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.804964 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.804965 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.805155 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.805520 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:22.805528 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:22.805766 (+ 238us) client_negotiation.cc:769] Sending connection context
0806 00:58:22.806637 (+ 871us) client_negotiation.cc:240] Negotiation successful
0806 00:58:22.806898 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":39,"thread_start_us":108,"threads_started":1}
I20260806 00:58:22.807236 31952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.792959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.23.242.2:44381)
0806 00:58:22.793275 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:22.793299 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:22.793944 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:22.794622 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:22.794623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:22.795146 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:22.795383 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.795388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.796200 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.796201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.797622 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.797624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.798202 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.798203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.798350 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.798999 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:22.799092 (+ 93us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:22.802271 (+ 3179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:22.804356 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.804357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.804357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.804716 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.804963 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.804964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.804965 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.805155 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.805520 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:22.805528 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:22.805766 (+ 238us) client_negotiation.cc:769] Sending connection context
0806 00:58:22.806861 (+ 1095us) client_negotiation.cc:240] Negotiation successful
0806 00:58:22.806962 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260806 00:58:22.807835 31950 heartbeater.cc:347] Connected to a master server at 127.23.242.61:41127
I20260806 00:58:22.807955 31950 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:22.808254 31950 heartbeater.cc:511] Master 127.23.242.61:41127 requested a full tablet report, sending...
I20260806 00:58:22.808799 31432 ts_manager.cc:194] Registered new tserver with Master: 4765fb0b489240bb920242b63a57e71f (127.23.242.2:45035)
I20260806 00:58:22.809180 31814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.793511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:55293 (local address 127.23.242.60:41799)
0806 00:58:22.793600 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.793603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.793614 (+ 11us) server_negotiation.cc:407] Connection header received
0806 00:58:22.794624 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.794627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.794642 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.794729 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.797161 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.797911 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.799112 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.799475 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.799559 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.805068 (+ 5509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.805070 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.805071 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.805080 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:22.807128 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.807978 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.807980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.807980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.808059 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.808333 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.808335 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.808335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.808491 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:22.808575 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:22.808890 (+ 315us) server_negotiation.cc:299] Negotiation successful
0806 00:58:22.809011 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 00:58:22.809995 31954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.793323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:41799 (local address 127.23.242.2:55293)
0806 00:58:22.793528 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:22.793540 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:22.793948 (+ 408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:22.794754 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:22.794755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:22.795153 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:22.795384 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.795390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.798011 (+ 2621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.798012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.798974 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.798977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.799588 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.799590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.799684 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.800359 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:22.800365 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:22.804935 (+ 4570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:22.807347 (+ 2412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.807348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.807349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.807864 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.808149 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.808150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.808150 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.808242 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.808682 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:22.808683 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:22.808738 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 00:58:22.809649 (+ 911us) client_negotiation.cc:240] Negotiation successful
0806 00:58:22.809798 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":5,"thread_start_us":76,"threads_started":1}
I20260806 00:58:22.810621 31948 heartbeater.cc:347] Connected to a master server at 127.23.242.60:41799
I20260806 00:58:22.810705 31948 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:22.810784 31948 heartbeater.cc:511] Master 127.23.242.60:41799 requested a full tablet report, sending...
I20260806 00:58:22.811103 31623 ts_manager.cc:194] Registered new tserver with Master: 4765fb0b489240bb920242b63a57e71f (127.23.242.2:45035)
I20260806 00:58:22.811318 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:39943
--local_ip_for_outbound_sockets=127.23.242.3
--tserver_master_addrs=127.23.242.62:38515
--webserver_port=44667
--webserver_interface=127.23.242.3
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.3.keytab
--principal=oryx/127.23.242.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.242.62:38515,127.23.242.61:41127
--tserver_master_addrs=127.23.242.62:38515,127.23.242.61:41127,127.23.242.60:41799 with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 00:58:22.811345 31949 heartbeater.cc:347] Connected to a master server at 127.23.242.62:38515
I20260806 00:58:22.811394 31949 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:22.811481 31949 heartbeater.cc:511] Master 127.23.242.62:38515 requested a full tablet report, sending...
I20260806 00:58:22.811844 30822 ts_manager.cc:194] Re-registered known tserver with Master: 4765fb0b489240bb920242b63a57e71f (127.23.242.2:45035)
I20260806 00:58:22.812427 30822 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.242.2@KRBTEST.COM'} at 127.23.242.2:44381
W20260806 00:58:22.912755 31955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:22.913182 31955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:22.913239 31955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:22.914726 31955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:22.914811 31955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:22.914872 31955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 00:58:22.916742 31955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.3.keytab
--principal=oryx/127.23.242.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:39943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=44667
--tserver_master_addrs=127.23.242.62:38515,127.23.242.61:41127,127.23.242.60:41799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:22.917119 31955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:22.917418 31955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:22.924091 31961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:22.924126 31963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:22.924126 31960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:22.925021 31955 server_base.cc:1034] running on GCE node
I20260806 00:58:22.925276 31955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:22.925530 31955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:22.926733 31955 hybrid_clock.cc:648] HybridClock initialized: now 1785977902926726 us; error 48 us; skew 500 ppm
Aug 06 00:58:22 dist-test-slave-dhph krb5kdc[30777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977902, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:22.930001 31955 init.cc:377] Logged in from keytab as oryx/127.23.242.3@KRBTEST.COM (short username oryx)
I20260806 00:58:22.930478 31955 webserver.cc:460] Webserver started at http://127.23.242.3:44667/ using document root <none> and password file <none>
I20260806 00:58:22.930794 31955 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:22.930892 31955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:22.932703 31955 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 00:58:22.933552 31971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:22.933820 31955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:22.933954 31955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "ec07cecb7fe24fed939834282411206d"
format_stamp: "Formatted at 2026-08-06 00:58:21 on dist-test-slave-dhph"
server_key: "055df28b4db838a8f4ef96f5ec85a6b1"
server_key_iv: "39259a08e3a56df0766146aa94e095c3"
server_key_version: "encryptionkey@0"
I20260806 00:58:22.934445 31955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:22.970832 31955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:22.973533 31955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:22.973721 31955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:22.973979 31955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:22.974327 31955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:22.974385 31955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:22.974431 31955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:22.974467 31955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:22.975695 31814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.972408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49528 (local address 127.23.242.60:41799)
0806 00:58:22.972473 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.972475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.972487 (+ 12us) server_negotiation.cc:407] Connection header received
0806 00:58:22.972558 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.972559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.972596 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.972675 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:22.972846 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.973370 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.974905 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.975327 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.975421 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.975489 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:22.975570 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 00:58:22.975619 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 00:58:22.975703 31977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.972280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:41799 (local address 127.0.0.1:49528)
0806 00:58:22.972402 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:22.972414 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:22.972482 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:22.972696 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:22.972697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:22.972699 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:22.972763 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.972770 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.973464 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.973465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.974795 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.974798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.975421 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.975422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.975490 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.975496 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 00:58:22.975564 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 00:58:22.975616 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260806 00:58:22.976301 31633 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260806 00:58:22.976531 31663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee13c8d77d44483c8dec48001f37be48. Latest consensus state: current_term: 1 leader_uuid: "ee13c8d77d44483c8dec48001f37be48" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: NON_VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: true } } }
I20260806 00:58:22.976647 31663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [sys.catalog]: This master's current role is: LEARNER
I20260806 00:58:22.981130 31955 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:39943
I20260806 00:58:22.981218 32087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:39943 every 8 connection(s)
I20260806 00:58:22.981570 31955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
I20260806 00:58:22.989215 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 31955
Aug 06 00:58:22 dist-test-slave-dhph krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977902, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.61@KRBTEST.COM
Aug 06 00:58:22 dist-test-slave-dhph krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977902, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM
Aug 06 00:58:22 dist-test-slave-dhph krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977902, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.60@KRBTEST.COM
I20260806 00:58:22.996413 30865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.982376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:47263 (local address 127.23.242.62:38515)
0806 00:58:22.982434 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.982437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.982806 (+ 369us) server_negotiation.cc:407] Connection header received
0806 00:58:22.983927 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.983928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.983940 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.983994 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.985048 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.985867 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.987363 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.987653 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.987716 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.992315 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.992317 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.992317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.992326 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:22.994190 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.994775 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.994777 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.994777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.994832 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.995464 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.995465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.995466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.995630 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:22.995712 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:22.996198 (+ 486us) server_negotiation.cc:299] Negotiation successful
0806 00:58:22.996296 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 00:58:22.996505 32092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.982358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:41799 (local address 127.23.242.3:50323)
0806 00:58:22.982711 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:22.982719 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:22.983799 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:22.984188 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:22.984188 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:22.984611 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:22.984940 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.984945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.986769 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.986771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.987417 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.987420 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.987989 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.987990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.988069 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.988612 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:22.988616 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:22.992571 (+ 3955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:22.994566 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.994566 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.994567 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.995319 (+ 752us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.995553 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.995554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.995555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.995614 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.995892 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:22.995893 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:22.995967 (+ 74us) client_negotiation.cc:769] Sending connection context
0806 00:58:22.996073 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 00:58:22.996261 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":9,"thread_start_us":112,"threads_started":1}
I20260806 00:58:22.996922 32093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.982357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.23.242.3:47263)
0806 00:58:22.982565 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:22.982711 (+ 146us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:22.983802 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:22.984086 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:22.984089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:22.984611 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:22.984940 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.984945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.985989 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.985990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.987269 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.987272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.987754 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.987755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.987901 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.988592 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:22.988604 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:22.992223 (+ 3619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:22.994291 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.994300 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.994303 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.994678 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.994910 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:22.994911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:22.994912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:22.995077 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:22.995749 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:22.995759 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:22.995968 (+ 209us) client_negotiation.cc:769] Sending connection context
0806 00:58:22.996073 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 00:58:22.996819 (+ 746us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":648,"thread_start_us":125,"threads_started":1}
I20260806 00:58:22.997298 32089 heartbeater.cc:347] Connected to a master server at 127.23.242.62:38515
I20260806 00:58:22.997416 32089 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:22.997503 31814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.982350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:50323 (local address 127.23.242.60:41799)
0806 00:58:22.982397 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.982399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.982806 (+ 407us) server_negotiation.cc:407] Connection header received
0806 00:58:22.984082 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.984084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.984096 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.984172 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.985987 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.986683 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.987502 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.987682 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.987829 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.992680 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.992681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.992681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.992688 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:22.994474 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.995414 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.995415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.995415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.995482 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.995686 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.995687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.995687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.995808 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:22.995865 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:22.997324 (+ 1459us) server_negotiation.cc:299] Negotiation successful
0806 00:58:22.997411 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 00:58:22.997643 32089 heartbeater.cc:511] Master 127.23.242.62:38515 requested a full tablet report, sending...
I20260806 00:58:22.997898 32088 heartbeater.cc:347] Connected to a master server at 127.23.242.60:41799
I20260806 00:58:22.997952 32088 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:22.998015 32088 heartbeater.cc:511] Master 127.23.242.60:41799 requested a full tablet report, sending...
I20260806 00:58:22.998386 31623 ts_manager.cc:194] Registered new tserver with Master: ec07cecb7fe24fed939834282411206d (127.23.242.3:39943)
I20260806 00:58:22.998536 30822 ts_manager.cc:194] Re-registered known tserver with Master: ec07cecb7fe24fed939834282411206d (127.23.242.3:39943)
I20260806 00:58:22.999084 30822 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.242.3@KRBTEST.COM'} at 127.23.242.3:47263
I20260806 00:58:23.000514 32100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.990623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33920 (local address 127.23.242.61:41127)
0806 00:58:22.990924 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.990926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.990935 (+ 9us) server_negotiation.cc:407] Connection header received
0806 00:58:22.990968 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.990969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.990983 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.991905 (+ 922us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.991944 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.992802 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.994041 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.994369 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.994752 (+ 383us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.996258 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.996260 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.996260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.996268 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:22.998619 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.999230 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.999232 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.999232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.999303 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:22.999550 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:22.999551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:22.999551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:22.999704 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:23.000243 (+ 539us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:23.000292 (+ 49us) server_negotiation.cc:299] Negotiation successful
0806 00:58:23.000409 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":64,"threads_started":1}
I20260806 00:58:23.002686 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.990963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49532 (local address 127.23.242.60:41799)
0806 00:58:22.994841 (+ 3878us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.994843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.994850 (+ 7us) server_negotiation.cc:407] Connection header received
0806 00:58:22.994873 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.994874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.994905 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.994996 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.995699 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.996393 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.997375 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.997648 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.997712 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.999177 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.999178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.999179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.999186 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:23.001383 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:23.001768 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:23.001769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:23.001770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:23.001835 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:23.002059 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:23.002060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:23.002060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:23.002205 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:23.002315 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:23.002456 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 00:58:23.002540 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3856,"thread_start_us":68,"threads_started":1}
I20260806 00:58:23.003743 31471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.982949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:41961 (local address 127.23.242.61:41127)
0806 00:58:22.982991 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.982993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.984063 (+ 1070us) server_negotiation.cc:407] Connection header received
0806 00:58:22.984095 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.984096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.984108 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.984173 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.985327 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.986141 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.987772 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.988111 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.988199 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:23.000593 (+ 12394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:23.000594 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:23.000594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:23.000601 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:23.002446 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:23.002881 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:23.002882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:23.002882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:23.002937 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:23.003152 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:23.003152 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:23.003153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:23.003283 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:23.003349 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:23.003542 (+ 193us) server_negotiation.cc:299] Negotiation successful
0806 00:58:23.003627 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260806 00:58:23.003882 32094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.982759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:41127 (local address 127.23.242.3:41961)
0806 00:58:22.983979 (+ 1220us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:22.983989 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:22.984040 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:22.985050 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:22.985050 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:22.985182 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:22.985235 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.985237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.986277 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.986278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.987269 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:22.987272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.988213 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.988214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:22.988316 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.988783 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:22.988788 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:23.000465 (+ 11677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:23.002537 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:23.002538 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:23.002538 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:23.002802 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:23.003022 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:23.003023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:23.003024 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:23.003091 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:23.003357 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:23.003358 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:23.003404 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 00:58:23.003483 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 00:58:23.003709 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1204,"mutex_wait_us":4,"thread_start_us":77,"threads_started":1}
I20260806 00:58:23.004195 32099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:22.990484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57574 (local address 127.23.242.62:38515)
0806 00:58:22.990864 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:22.990867 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:22.991054 (+ 187us) server_negotiation.cc:407] Connection header received
0806 00:58:22.991217 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:22.991218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:22.991232 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:22.991284 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:22.992069 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.992988 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.994097 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:22.994452 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:22.994501 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:22.999767 (+ 5266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:22.999769 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:22.999769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:22.999776 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:23.002298 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:23.002937 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:23.002938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:23.002938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:23.003008 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:23.003244 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:23.003245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:23.003245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:23.003387 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:23.003574 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:23.003655 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 00:58:23.003749 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":77,"threads_started":1}
I20260806 00:58:23.004673 32090 heartbeater.cc:347] Connected to a master server at 127.23.242.61:41127
I20260806 00:58:23.004774 32090 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:23.004889 32090 heartbeater.cc:511] Master 127.23.242.61:41127 requested a full tablet report, sending...
I20260806 00:58:23.006187 31432 ts_manager.cc:194] Registered new tserver with Master: ec07cecb7fe24fed939834282411206d (127.23.242.3:39943)
I20260806 00:58:23.007190 30822 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 00:58:23.009286 30822 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 00:58:23.010303 31976 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee13c8d77d44483c8dec48001f37be48" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: NON_VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: true } } }
I20260806 00:58:23.010432 31976 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:23.012544 31976 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48: attempting to promote NON_VOTER 12658ab315ca4652baab3a5f69d785e3 to VOTER
I20260806 00:58:23.012778 31976 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: false } }
I20260806 00:58:23.013415 31633 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 1 LEARNER]: Refusing update from remote peer ee13c8d77d44483c8dec48001f37be48: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 00:58:23.013420 31442 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 1 FOLLOWER]: Refusing update from remote peer ee13c8d77d44483c8dec48001f37be48: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 00:58:23.013684 32107 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 00:58:23.014181 32108 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 00:58:23.015556 32108 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 12658ab315ca4652baab3a5f69d785e3 (127.23.242.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: false } } }
I20260806 00:58:23.015785 31442 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 12658ab315ca4652baab3a5f69d785e3 (127.23.242.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: false } } }
I20260806 00:58:23.015774 31633 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 12658ab315ca4652baab3a5f69d785e3 (127.23.242.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: false } } }
I20260806 00:58:23.016700 32108 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ee13c8d77d44483c8dec48001f37be48" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: false } } }
I20260806 00:58:23.016788 32108 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:23.016842 32108 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee13c8d77d44483c8dec48001f37be48" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: false } } }
I20260806 00:58:23.016808 31663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee13c8d77d44483c8dec48001f37be48" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: false } } }
I20260806 00:58:23.016889 32108 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:23.016911 31663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 00:58:23.016930 32107 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee13c8d77d44483c8dec48001f37be48" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: false } } }
I20260806 00:58:23.017023 32107 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:23.018220 32117 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee13c8d77d44483c8dec48001f37be48" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: false } } }
I20260806 00:58:23.018340 32117 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 00:58:23.023718 31977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:23.017404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:45035 (local address 127.0.0.1:36268)
0806 00:58:23.017910 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:23.017950 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:23.018041 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:23.018929 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:23.018930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:23.018932 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:23.018992 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:23.018994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.021251 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.021252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:23.022289 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:23.022292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.023471 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.023472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:23.023533 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:23.023537 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 00:58:23.023585 (+ 48us) client_negotiation.cc:240] Negotiation successful
0806 00:58:23.023635 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470}
I20260806 00:58:23.023953 32120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:23.017517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36268 (local address 127.23.242.2:45035)
0806 00:58:23.018717 (+ 1200us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:23.018722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:23.018736 (+ 14us) server_negotiation.cc:407] Connection header received
0806 00:58:23.018786 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:23.018789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:23.018808 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:23.018911 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:23.019063 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.020863 (+ 1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.022394 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.023363 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.023479 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:23.023611 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:23.023645 (+ 34us) server_negotiation.cc:299] Negotiation successful
0806 00:58:23.023696 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1148,"thread_start_us":290,"threads_started":1}
I20260806 00:58:23.024318 32118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:23.017504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:42435 (local address 127.0.0.1:56926)
0806 00:58:23.018857 (+ 1353us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:23.018868 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:23.018994 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:23.019707 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:23.019708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:23.019710 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:23.019754 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:23.019756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.021754 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.021756 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:23.022996 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:23.022999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.024070 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.024072 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:23.024140 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:23.024146 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 00:58:23.024196 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 00:58:23.024245 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1330,"thread_start_us":306,"threads_started":1}
I20260806 00:58:23.024395 32119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:23.017487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:39943 (local address 127.0.0.1:36524)
0806 00:58:23.018667 (+ 1180us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:23.018684 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:23.018790 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:23.019140 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:23.019141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:23.019144 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:23.019196 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:23.019198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.020932 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.020933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:23.022372 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:23.022375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.024072 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.024073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:23.024140 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:23.024146 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 00:58:23.024250 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 00:58:23.024288 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1146,"mutex_wait_us":46,"thread_start_us":126,"threads_started":1}
I20260806 00:58:23.024574 32122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:23.018285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56926 (local address 127.23.242.1:42435)
0806 00:58:23.019457 (+ 1172us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:23.019462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:23.019474 (+ 12us) server_negotiation.cc:407] Connection header received
0806 00:58:23.019517 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:23.019519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:23.019537 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:23.019622 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:23.019835 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.021626 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.023146 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.023888 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.023954 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:23.024033 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:23.024451 (+ 418us) server_negotiation.cc:299] Negotiation successful
0806 00:58:23.024503 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1067,"thread_start_us":97,"threads_started":1}
I20260806 00:58:23.024768 32121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:23.018114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36524 (local address 127.23.242.3:39943)
0806 00:58:23.018381 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:23.018388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:23.018783 (+ 395us) server_negotiation.cc:407] Connection header received
0806 00:58:23.018907 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:23.018909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:23.018926 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:23.019002 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:23.019306 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.020799 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.022496 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.023960 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.024069 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:23.024157 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:23.024272 (+ 115us) server_negotiation.cc:299] Negotiation successful
0806 00:58:23.024322 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":144,"threads_started":1}
I20260806 00:58:23.024978 31881 tablet_service.cc:1467] Processing CreateTablet for tablet 333edef21b1a4bd59551303f11ebd3c9 (DEFAULT_TABLE table=test-table [id=83b498efd2c7466188080fdec3b42d93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:23.025178 31738 tablet_service.cc:1467] Processing CreateTablet for tablet 333edef21b1a4bd59551303f11ebd3c9 (DEFAULT_TABLE table=test-table [id=83b498efd2c7466188080fdec3b42d93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:23.025444 31738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 333edef21b1a4bd59551303f11ebd3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:23.025429 31881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 333edef21b1a4bd59551303f11ebd3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:23.025444 32021 tablet_service.cc:1467] Processing CreateTablet for tablet 333edef21b1a4bd59551303f11ebd3c9 (DEFAULT_TABLE table=test-table [id=83b498efd2c7466188080fdec3b42d93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:23.025956 32021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 333edef21b1a4bd59551303f11ebd3c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:23.028301 32123 tablet_bootstrap.cc:492] T 333edef21b1a4bd59551303f11ebd3c9 P ba9ec1e1be664d3b9e705abaf8f50594: Bootstrap starting.
I20260806 00:58:23.029002 32123 tablet_bootstrap.cc:654] T 333edef21b1a4bd59551303f11ebd3c9 P ba9ec1e1be664d3b9e705abaf8f50594: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:23.029249 32123 log.cc:824] T 333edef21b1a4bd59551303f11ebd3c9 P ba9ec1e1be664d3b9e705abaf8f50594: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:23.029592 32124 tablet_bootstrap.cc:492] T 333edef21b1a4bd59551303f11ebd3c9 P 4765fb0b489240bb920242b63a57e71f: Bootstrap starting.
I20260806 00:58:23.029968 32123 tablet_bootstrap.cc:492] T 333edef21b1a4bd59551303f11ebd3c9 P ba9ec1e1be664d3b9e705abaf8f50594: No bootstrap required, opened a new log
I20260806 00:58:23.030054 32123 ts_tablet_manager.cc:1397] T 333edef21b1a4bd59551303f11ebd3c9 P ba9ec1e1be664d3b9e705abaf8f50594: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 00:58:23.030319 32126 tablet_bootstrap.cc:492] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d: Bootstrap starting.
I20260806 00:58:23.031076 32126 tablet_bootstrap.cc:654] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:23.031338 32124 tablet_bootstrap.cc:654] T 333edef21b1a4bd59551303f11ebd3c9 P 4765fb0b489240bb920242b63a57e71f: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:23.031491 32126 log.cc:824] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:23.032040 32124 log.cc:824] T 333edef21b1a4bd59551303f11ebd3c9 P 4765fb0b489240bb920242b63a57e71f: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:23.032681 32123 raft_consensus.cc:348] T 333edef21b1a4bd59551303f11ebd3c9 P ba9ec1e1be664d3b9e705abaf8f50594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765fb0b489240bb920242b63a57e71f" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45035 } } peers { permanent_uuid: "ba9ec1e1be664d3b9e705abaf8f50594" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42435 } } peers { permanent_uuid: "ec07cecb7fe24fed939834282411206d" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39943 } }
I20260806 00:58:23.032824 32126 tablet_bootstrap.cc:492] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d: No bootstrap required, opened a new log
I20260806 00:58:23.032858 32123 raft_consensus.cc:374] T 333edef21b1a4bd59551303f11ebd3c9 P ba9ec1e1be664d3b9e705abaf8f50594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:23.032878 32126 ts_tablet_manager.cc:1397] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260806 00:58:23.032928 32123 raft_consensus.cc:729] T 333edef21b1a4bd59551303f11ebd3c9 P ba9ec1e1be664d3b9e705abaf8f50594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba9ec1e1be664d3b9e705abaf8f50594, State: Initialized, Role: FOLLOWER
I20260806 00:58:23.033069 32123 consensus_queue.cc:260] T 333edef21b1a4bd59551303f11ebd3c9 P ba9ec1e1be664d3b9e705abaf8f50594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765fb0b489240bb920242b63a57e71f" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45035 } } peers { permanent_uuid: "ba9ec1e1be664d3b9e705abaf8f50594" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42435 } } peers { permanent_uuid: "ec07cecb7fe24fed939834282411206d" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39943 } }
I20260806 00:58:23.033699 31809 heartbeater.cc:503] Master 127.23.242.62:38515 was elected leader, sending a full tablet report...
I20260806 00:58:23.033690 32123 ts_tablet_manager.cc:1428] T 333edef21b1a4bd59551303f11ebd3c9 P ba9ec1e1be664d3b9e705abaf8f50594: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260806 00:58:23.035259 32124 tablet_bootstrap.cc:492] T 333edef21b1a4bd59551303f11ebd3c9 P 4765fb0b489240bb920242b63a57e71f: No bootstrap required, opened a new log
I20260806 00:58:23.035332 32124 ts_tablet_manager.cc:1397] T 333edef21b1a4bd59551303f11ebd3c9 P 4765fb0b489240bb920242b63a57e71f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260806 00:58:23.035333 32126 raft_consensus.cc:348] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765fb0b489240bb920242b63a57e71f" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45035 } } peers { permanent_uuid: "ba9ec1e1be664d3b9e705abaf8f50594" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42435 } } peers { permanent_uuid: "ec07cecb7fe24fed939834282411206d" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39943 } }
I20260806 00:58:23.035533 32126 raft_consensus.cc:374] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:23.035585 32126 raft_consensus.cc:729] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec07cecb7fe24fed939834282411206d, State: Initialized, Role: FOLLOWER
I20260806 00:58:23.035722 32126 consensus_queue.cc:260] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765fb0b489240bb920242b63a57e71f" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45035 } } peers { permanent_uuid: "ba9ec1e1be664d3b9e705abaf8f50594" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42435 } } peers { permanent_uuid: "ec07cecb7fe24fed939834282411206d" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39943 } }
I20260806 00:58:23.036008 32126 ts_tablet_manager.cc:1428] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260806 00:58:23.036726 32089 heartbeater.cc:503] Master 127.23.242.62:38515 was elected leader, sending a full tablet report...
I20260806 00:58:23.038889 32124 raft_consensus.cc:348] T 333edef21b1a4bd59551303f11ebd3c9 P 4765fb0b489240bb920242b63a57e71f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765fb0b489240bb920242b63a57e71f" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45035 } } peers { permanent_uuid: "ba9ec1e1be664d3b9e705abaf8f50594" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42435 } } peers { permanent_uuid: "ec07cecb7fe24fed939834282411206d" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39943 } }
I20260806 00:58:23.039084 32124 raft_consensus.cc:374] T 333edef21b1a4bd59551303f11ebd3c9 P 4765fb0b489240bb920242b63a57e71f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:23.039127 32124 raft_consensus.cc:729] T 333edef21b1a4bd59551303f11ebd3c9 P 4765fb0b489240bb920242b63a57e71f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4765fb0b489240bb920242b63a57e71f, State: Initialized, Role: FOLLOWER
I20260806 00:58:23.039266 32124 consensus_queue.cc:260] T 333edef21b1a4bd59551303f11ebd3c9 P 4765fb0b489240bb920242b63a57e71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765fb0b489240bb920242b63a57e71f" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45035 } } peers { permanent_uuid: "ba9ec1e1be664d3b9e705abaf8f50594" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42435 } } peers { permanent_uuid: "ec07cecb7fe24fed939834282411206d" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39943 } }
I20260806 00:58:23.039533 32124 ts_tablet_manager.cc:1428] T 333edef21b1a4bd59551303f11ebd3c9 P 4765fb0b489240bb920242b63a57e71f: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260806 00:58:23.039667 31949 heartbeater.cc:503] Master 127.23.242.62:38515 was elected leader, sending a full tablet report...
W20260806 00:58:23.042903 31951 tablet.cc:2369] T 333edef21b1a4bd59551303f11ebd3c9 P 4765fb0b489240bb920242b63a57e71f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 00:58:23.131490 31811 tablet.cc:2369] T 333edef21b1a4bd59551303f11ebd3c9 P ba9ec1e1be664d3b9e705abaf8f50594: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 00:58:23.200963 32130 raft_consensus.cc:482] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 00:58:23.201083 32130 raft_consensus.cc:504] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765fb0b489240bb920242b63a57e71f" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45035 } } peers { permanent_uuid: "ba9ec1e1be664d3b9e705abaf8f50594" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42435 } } peers { permanent_uuid: "ec07cecb7fe24fed939834282411206d" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39943 } }
I20260806 00:58:23.201802 32130 leader_election.cc:288] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4765fb0b489240bb920242b63a57e71f (127.23.242.2:45035), ba9ec1e1be664d3b9e705abaf8f50594 (127.23.242.1:42435)
I20260806 00:58:23.205140 32120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:23.202149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:53409 (local address 127.23.242.2:45035)
0806 00:58:23.202230 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:23.202232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:23.202243 (+ 11us) server_negotiation.cc:407] Connection header received
0806 00:58:23.202296 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:23.202297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:23.202310 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:23.202410 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:23.202646 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.203291 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.204269 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.204734 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.204799 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:23.204806 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:23.204988 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 00:58:23.205038 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 00:58:23.205165 32094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:23.202016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:45035 (local address 127.23.242.3:53409)
0806 00:58:23.202081 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:23.202094 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:23.202143 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:23.202420 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:23.202421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:23.202422 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:23.202499 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:23.202503 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.203383 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.203384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:23.204179 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:23.204181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.204833 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.204834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:23.204920 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:23.204924 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 00:58:23.204979 (+ 55us) client_negotiation.cc:240] Negotiation successful
0806 00:58:23.205029 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260806 00:58:23.205246 32122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:23.202326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:49409 (local address 127.23.242.1:42435)
0806 00:58:23.202395 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:23.202398 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:23.202405 (+ 7us) server_negotiation.cc:407] Connection header received
0806 00:58:23.202431 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:23.202432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:23.202441 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:23.202498 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:23.202770 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.203220 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.204391 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.204851 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.204930 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:23.204939 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:23.205127 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 00:58:23.205168 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 00:58:23.205514 32093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:23.202192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:42435 (local address 127.23.242.3:49409)
0806 00:58:23.202238 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:23.202247 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:23.202280 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:23.202518 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:23.202519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:23.202521 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:23.202644 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:23.202646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.203306 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.203307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:23.204280 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:23.204284 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.204948 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.204949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:23.205042 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:23.205046 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 00:58:23.205112 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 00:58:23.205155 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":1}
I20260806 00:58:23.205606 31901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "333edef21b1a4bd59551303f11ebd3c9" candidate_uuid: "ec07cecb7fe24fed939834282411206d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4765fb0b489240bb920242b63a57e71f" is_pre_election: true
I20260806 00:58:23.205799 31901 raft_consensus.cc:2454] T 333edef21b1a4bd59551303f11ebd3c9 P 4765fb0b489240bb920242b63a57e71f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec07cecb7fe24fed939834282411206d in term 0.
I20260806 00:58:23.205858 31760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "333edef21b1a4bd59551303f11ebd3c9" candidate_uuid: "ec07cecb7fe24fed939834282411206d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba9ec1e1be664d3b9e705abaf8f50594" is_pre_election: true
I20260806 00:58:23.206000 31760 raft_consensus.cc:2454] T 333edef21b1a4bd59551303f11ebd3c9 P ba9ec1e1be664d3b9e705abaf8f50594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec07cecb7fe24fed939834282411206d in term 0.
I20260806 00:58:23.206251 31972 leader_election.cc:302] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba9ec1e1be664d3b9e705abaf8f50594, ec07cecb7fe24fed939834282411206d; no voters:
I20260806 00:58:23.206395 32130 raft_consensus.cc:2781] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 00:58:23.206452 32130 raft_consensus.cc:482] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 00:58:23.206483 32130 raft_consensus.cc:3037] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:23.207284 32130 raft_consensus.cc:504] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765fb0b489240bb920242b63a57e71f" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45035 } } peers { permanent_uuid: "ba9ec1e1be664d3b9e705abaf8f50594" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42435 } } peers { permanent_uuid: "ec07cecb7fe24fed939834282411206d" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39943 } }
I20260806 00:58:23.207438 32130 leader_election.cc:288] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d [CANDIDATE]: Term 1 election: Requested vote from peers 4765fb0b489240bb920242b63a57e71f (127.23.242.2:45035), ba9ec1e1be664d3b9e705abaf8f50594 (127.23.242.1:42435)
I20260806 00:58:23.207647 31901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "333edef21b1a4bd59551303f11ebd3c9" candidate_uuid: "ec07cecb7fe24fed939834282411206d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4765fb0b489240bb920242b63a57e71f"
I20260806 00:58:23.207752 31901 raft_consensus.cc:3037] T 333edef21b1a4bd59551303f11ebd3c9 P 4765fb0b489240bb920242b63a57e71f [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:23.207721 31760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "333edef21b1a4bd59551303f11ebd3c9" candidate_uuid: "ec07cecb7fe24fed939834282411206d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba9ec1e1be664d3b9e705abaf8f50594"
I20260806 00:58:23.207862 31760 raft_consensus.cc:3037] T 333edef21b1a4bd59551303f11ebd3c9 P ba9ec1e1be664d3b9e705abaf8f50594 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:23.208487 31901 raft_consensus.cc:2454] T 333edef21b1a4bd59551303f11ebd3c9 P 4765fb0b489240bb920242b63a57e71f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec07cecb7fe24fed939834282411206d in term 1.
I20260806 00:58:23.208673 31972 leader_election.cc:302] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4765fb0b489240bb920242b63a57e71f, ec07cecb7fe24fed939834282411206d; no voters:
I20260806 00:58:23.208745 31760 raft_consensus.cc:2454] T 333edef21b1a4bd59551303f11ebd3c9 P ba9ec1e1be664d3b9e705abaf8f50594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec07cecb7fe24fed939834282411206d in term 1.
I20260806 00:58:23.208804 32130 raft_consensus.cc:2781] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:23.209012 32130 raft_consensus.cc:686] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d [term 1 LEADER]: Becoming Leader. State: Replica: ec07cecb7fe24fed939834282411206d, State: Running, Role: LEADER
I20260806 00:58:23.209131 32130 consensus_queue.cc:237] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765fb0b489240bb920242b63a57e71f" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45035 } } peers { permanent_uuid: "ba9ec1e1be664d3b9e705abaf8f50594" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42435 } } peers { permanent_uuid: "ec07cecb7fe24fed939834282411206d" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39943 } }
I20260806 00:58:23.210144 30822 catalog_manager.cc:5495] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d reported cstate change: term changed from 0 to 1, leader changed from <none> to ec07cecb7fe24fed939834282411206d (127.23.242.3). New cstate: current_term: 1 leader_uuid: "ec07cecb7fe24fed939834282411206d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4765fb0b489240bb920242b63a57e71f" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba9ec1e1be664d3b9e705abaf8f50594" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec07cecb7fe24fed939834282411206d" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39943 } health_report { overall_health: HEALTHY } } }
W20260806 00:58:23.233130 32091 tablet.cc:2369] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 00:58:23.256946 32121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:23.253903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36534 (local address 127.23.242.3:39943)
0806 00:58:23.253986 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:23.253988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:23.253996 (+ 8us) server_negotiation.cc:407] Connection header received
0806 00:58:23.254036 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:23.254037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:23.254057 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:23.254348 (+ 291us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 00:58:23.254480 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.255079 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.256089 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.256296 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.256390 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:23.256573 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 00:58:23.256690 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 00:58:23.256780 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:23.256826 (+ 46us) server_negotiation.cc:299] Negotiation successful
0806 00:58:23.256850 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 00:58:23.260116 31901 raft_consensus.cc:1261] T 333edef21b1a4bd59551303f11ebd3c9 P 4765fb0b489240bb920242b63a57e71f [term 1 FOLLOWER]: Refusing update from remote peer ec07cecb7fe24fed939834282411206d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:23.260145 31760 raft_consensus.cc:1261] T 333edef21b1a4bd59551303f11ebd3c9 P ba9ec1e1be664d3b9e705abaf8f50594 [term 1 FOLLOWER]: Refusing update from remote peer ec07cecb7fe24fed939834282411206d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:23.260772 32132 consensus_queue.cc:1035] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba9ec1e1be664d3b9e705abaf8f50594" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:23.261229 32130 consensus_queue.cc:1035] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4765fb0b489240bb920242b63a57e71f" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:23.264851 32135 mvcc.cc:205] Tried to move back new op lower bound from 7315365491749621760 to 7315365491545583616. Current Snapshot: MvccSnapshot[applied={T|T < 7315365491749621760}]
I20260806 00:58:23.265791 32137 mvcc.cc:205] Tried to move back new op lower bound from 7315365491749621760 to 7315365491545583616. Current Snapshot: MvccSnapshot[applied={T|T < 7315365491749621760}]
I20260806 00:58:23.268886 30821 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:58:23.269021 30821 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:58:23.271283 30821 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee13c8d77d44483c8dec48001f37be48: Sending DeleteTablet for 3 replicas of tablet 333edef21b1a4bd59551303f11ebd3c9
I20260806 00:58:23.271843 31738 tablet_service.cc:1514] Processing DeleteTablet for tablet 333edef21b1a4bd59551303f11ebd3c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:23 UTC) from {username='oryx', principal='oryx/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:56926
I20260806 00:58:23.271854 31881 tablet_service.cc:1514] Processing DeleteTablet for tablet 333edef21b1a4bd59551303f11ebd3c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:23 UTC) from {username='oryx', principal='oryx/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:36268
I20260806 00:58:23.272621 32021 tablet_service.cc:1514] Processing DeleteTablet for tablet 333edef21b1a4bd59551303f11ebd3c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:23 UTC) from {username='oryx', principal='oryx/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:36524
I20260806 00:58:23.270444 32136 mvcc.cc:205] Tried to move back new op lower bound from 7315365491749621760 to 7315365491545583616. Current Snapshot: MvccSnapshot[applied={T|T < 7315365491749621760 or (T in {7315365491749621760})}]
I20260806 00:58:23.273265 32146 tablet_replica.cc:330] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d: stopping tablet replica
I20260806 00:58:23.273367 32144 tablet_replica.cc:330] T 333edef21b1a4bd59551303f11ebd3c9 P ba9ec1e1be664d3b9e705abaf8f50594: stopping tablet replica
I20260806 00:58:23.273490 32144 raft_consensus.cc:2229] T 333edef21b1a4bd59551303f11ebd3c9 P ba9ec1e1be664d3b9e705abaf8f50594 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:23.273625 32144 raft_consensus.cc:2258] T 333edef21b1a4bd59551303f11ebd3c9 P ba9ec1e1be664d3b9e705abaf8f50594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:23.273691 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 30790
I20260806 00:58:23.273809 32145 tablet_replica.cc:330] T 333edef21b1a4bd59551303f11ebd3c9 P 4765fb0b489240bb920242b63a57e71f: stopping tablet replica
I20260806 00:58:23.273926 32145 raft_consensus.cc:2229] T 333edef21b1a4bd59551303f11ebd3c9 P 4765fb0b489240bb920242b63a57e71f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:23.274044 32145 raft_consensus.cc:2258] T 333edef21b1a4bd59551303f11ebd3c9 P 4765fb0b489240bb920242b63a57e71f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:23.274614 32146 raft_consensus.cc:2229] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d [term 1 LEADER]: Raft consensus shutting down.
I20260806 00:58:23.274747 32146 raft_consensus.cc:2258] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:23.277096 32144 ts_tablet_manager.cc:1905] T 333edef21b1a4bd59551303f11ebd3c9 P ba9ec1e1be664d3b9e705abaf8f50594: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:23.277323 32146 ts_tablet_manager.cc:1905] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:23.279709 32146 ts_tablet_manager.cc:1918] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 00:58:23.279819 32146 log.cc:1196] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/wal/wals/333edef21b1a4bd59551303f11ebd3c9
I20260806 00:58:23.280165 32146 ts_tablet_manager.cc:1939] T 333edef21b1a4bd59551303f11ebd3c9 P ec07cecb7fe24fed939834282411206d: Deleting consensus metadata
I20260806 00:58:23.280599 32144 ts_tablet_manager.cc:1918] T 333edef21b1a4bd59551303f11ebd3c9 P ba9ec1e1be664d3b9e705abaf8f50594: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 00:58:23.280679 32144 log.cc:1196] T 333edef21b1a4bd59551303f11ebd3c9 P ba9ec1e1be664d3b9e705abaf8f50594: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-0/wal/wals/333edef21b1a4bd59551303f11ebd3c9
I20260806 00:58:23.280984 32144 ts_tablet_manager.cc:1939] T 333edef21b1a4bd59551303f11ebd3c9 P ba9ec1e1be664d3b9e705abaf8f50594: Deleting consensus metadata
W20260806 00:58:23.285118 31693 connection.cc:535] server connection from 127.0.0.1:56926 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 00:58:23.285291 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:23.272978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49544 (local address 127.23.242.60:41799)
0806 00:58:23.273158 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:23.273161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:23.273540 (+ 379us) server_negotiation.cc:407] Connection header received
0806 00:58:23.274899 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:23.274901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:23.274919 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:23.274956 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:23.275905 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.276807 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.280191 (+ 3384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.280492 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.280545 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:23.282163 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:23.282165 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:23.282165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:23.282173 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:23.283763 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:23.284155 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:23.284156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:23.284157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:23.284218 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:23.284430 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:23.284431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:23.284431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:23.284558 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:23.284734 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:23.284988 (+ 254us) server_negotiation.cc:299] Negotiation successful
0806 00:58:23.285170 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
W20260806 00:58:23.285715 31975 connection.cc:535] server connection from 127.0.0.1:36524 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 00:58:23.287357 32145 ts_tablet_manager.cc:1905] T 333edef21b1a4bd59551303f11ebd3c9 P 4765fb0b489240bb920242b63a57e71f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:23.289706 32145 ts_tablet_manager.cc:1918] T 333edef21b1a4bd59551303f11ebd3c9 P 4765fb0b489240bb920242b63a57e71f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 00:58:23.289831 32145 log.cc:1196] T 333edef21b1a4bd59551303f11ebd3c9 P 4765fb0b489240bb920242b63a57e71f: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/wal/wals/333edef21b1a4bd59551303f11ebd3c9
I20260806 00:58:23.290215 32145 ts_tablet_manager.cc:1939] T 333edef21b1a4bd59551303f11ebd3c9 P 4765fb0b489240bb920242b63a57e71f: Deleting consensus metadata
W20260806 00:58:23.291944 31834 connection.cc:441] server connection from 127.0.0.1:36268 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36268 (request call id 1) could send its response
I20260806 00:58:23.299273 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:23.288352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49554 (local address 127.23.242.60:41799)
0806 00:58:23.288389 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:23.288391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:23.288526 (+ 135us) server_negotiation.cc:407] Connection header received
0806 00:58:23.288629 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:23.288630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:23.288641 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:23.288672 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:23.289740 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.290539 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.291933 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.292288 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.292365 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:23.294168 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:23.294169 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:23.294170 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:23.294178 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:23.296677 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:23.297764 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:23.297765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:23.297766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:23.297830 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:23.298321 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:23.298322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:23.298323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:23.298506 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:23.298670 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:23.298943 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 00:58:23.299042 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 00:58:23.299618 31471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:23.288034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33932 (local address 127.23.242.61:41127)
0806 00:58:23.288084 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:23.288087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:23.288526 (+ 439us) server_negotiation.cc:407] Connection header received
0806 00:58:23.288678 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:23.288679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:23.288691 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:23.288721 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:23.290819 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.291819 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.293133 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:23.293480 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:23.293535 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:23.295356 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:23.295357 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:23.295357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:23.295366 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:23.296960 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:23.298146 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:23.298147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:23.298148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:23.298264 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:23.298941 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:23.298942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:23.298942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:23.299088 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:23.299202 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:23.299411 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 00:58:23.299493 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 00:58:24.268426 32155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:24.268003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.23.242.2:35411)
0806 00:58:24.268231 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:24.268345 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":140,"threads_started":1}
W20260806 00:58:24.268841 31949 heartbeater.cc:644] Failed to heartbeat to 127.23.242.62:38515 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
I20260806 00:58:24.269239 32156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:24.268990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.23.242.3:46779)
0806 00:58:24.269105 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:24.269187 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":64,"threads_started":1}
W20260806 00:58:24.269450 32089 heartbeater.cc:644] Failed to heartbeat to 127.23.242.62:38515 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
I20260806 00:58:24.273574 32157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:24.273305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.23.242.1:58731)
0806 00:58:24.273477 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:24.273533 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":61,"threads_started":1}
W20260806 00:58:24.273808 31809 heartbeater.cc:644] Failed to heartbeat to 127.23.242.62:38515 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
I20260806 00:58:24.771528 32158 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee13c8d77d44483c8dec48001f37be48)
I20260806 00:58:24.771715 32158 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: false } }
I20260806 00:58:24.771842 32159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee13c8d77d44483c8dec48001f37be48)
I20260806 00:58:24.771974 32159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: false } }
I20260806 00:58:24.772106 32158 leader_election.cc:288] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee13c8d77d44483c8dec48001f37be48 (127.23.242.62:38515), 12658ab315ca4652baab3a5f69d785e3 (127.23.242.60:41799)
I20260806 00:58:24.772466 32161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:24.772248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.0.0.1:58062)
0806 00:58:24.772371 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:24.772420 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":51,"threads_started":1}
W20260806 00:58:24.772591 31419 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
I20260806 00:58:24.772750 32159 leader_election.cc:288] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee13c8d77d44483c8dec48001f37be48 (127.23.242.62:38515), a6ca6242d6744e37a9d2b1aa76faef3c (127.23.242.61:41127)
I20260806 00:58:24.772960 32164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:24.772580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.0.0.1:58070)
0806 00:58:24.772867 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:24.772921 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":37,"threads_started":1}
I20260806 00:58:24.773022 32161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:24.772940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.0.0.1:58074)
0806 00:58:24.772974 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:24.772998 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260806 00:58:24.773065 31609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
W20260806 00:58:24.773103 31419 leader_election.cc:334] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ee13c8d77d44483c8dec48001f37be48 (127.23.242.62:38515): Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
I20260806 00:58:24.773520 32164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:24.773413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.0.0.1:58090)
0806 00:58:24.773465 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:24.773496 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260806 00:58:24.773656 31609 leader_election.cc:334] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ee13c8d77d44483c8dec48001f37be48 (127.23.242.62:38515): Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
I20260806 00:58:24.775815 32160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:24.772264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:41799 (local address 127.0.0.1:49566)
0806 00:58:24.772456 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:24.772468 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:24.772693 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:24.772996 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:24.772997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:24.773000 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:24.773266 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:24.773269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:24.774137 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:24.774138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:24.775002 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:24.775004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:24.775552 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:24.775553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:24.775621 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:24.775625 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 00:58:24.775701 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 00:58:24.775745 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":6,"thread_start_us":38,"threads_started":1}
I20260806 00:58:24.775830 32162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:24.772427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49566 (local address 127.23.242.60:41799)
0806 00:58:24.772691 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:24.772693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:24.772704 (+ 11us) server_negotiation.cc:407] Connection header received
0806 00:58:24.772868 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:24.772869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:24.772883 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:24.772942 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:24.773404 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:24.774041 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:24.775106 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:24.775466 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:24.775551 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:24.775562 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:24.775706 (+ 144us) server_negotiation.cc:299] Negotiation successful
0806 00:58:24.775745 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":54,"threads_started":1}
I20260806 00:58:24.776163 32165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:24.772680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33948 (local address 127.23.242.61:41127)
0806 00:58:24.772854 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:24.772856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:24.772866 (+ 10us) server_negotiation.cc:407] Connection header received
0806 00:58:24.772892 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:24.772893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:24.772905 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:24.773169 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:24.773205 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:24.773925 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:24.775067 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:24.775445 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:24.775524 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:24.775535 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:24.776052 (+ 517us) server_negotiation.cc:299] Negotiation successful
0806 00:58:24.776097 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":157,"thread_start_us":48,"threads_started":1}
I20260806 00:58:24.776207 32163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:24.772547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:41127 (local address 127.0.0.1:33948)
0806 00:58:24.772747 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:24.772756 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:24.772802 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:24.773015 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:24.773016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:24.773019 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:24.773083 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:24.773085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:24.774008 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:24.774009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:24.774980 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:24.774982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:24.775523 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:24.775524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:24.775637 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:24.775642 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 00:58:24.775941 (+ 299us) client_negotiation.cc:240] Negotiation successful
0806 00:58:24.776003 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":57,"threads_started":1}
I20260806 00:58:24.776175 31633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "12658ab315ca4652baab3a5f69d785e3" is_pre_election: true
I20260806 00:58:24.776331 31633 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6ca6242d6744e37a9d2b1aa76faef3c in term 1.
I20260806 00:58:24.776556 31418 leader_election.cc:302] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12658ab315ca4652baab3a5f69d785e3, a6ca6242d6744e37a9d2b1aa76faef3c; no voters: ee13c8d77d44483c8dec48001f37be48
I20260806 00:58:24.776572 31442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12658ab315ca4652baab3a5f69d785e3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" is_pre_election: true
I20260806 00:58:24.776675 31442 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12658ab315ca4652baab3a5f69d785e3 in term 1.
I20260806 00:58:24.776724 32158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 00:58:24.776767 32158 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee13c8d77d44483c8dec48001f37be48)
I20260806 00:58:24.776798 32158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 1 FOLLOWER]: Advancing to term 2
I20260806 00:58:24.776878 31608 leader_election.cc:302] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12658ab315ca4652baab3a5f69d785e3, a6ca6242d6744e37a9d2b1aa76faef3c; no voters: ee13c8d77d44483c8dec48001f37be48
I20260806 00:58:24.776995 32159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 00:58:24.777031 32159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee13c8d77d44483c8dec48001f37be48)
I20260806 00:58:24.777069 32159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 1 FOLLOWER]: Advancing to term 2
I20260806 00:58:24.777631 32158 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: false } }
I20260806 00:58:24.777776 32158 leader_election.cc:288] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [CANDIDATE]: Term 2 election: Requested vote from peers ee13c8d77d44483c8dec48001f37be48 (127.23.242.62:38515), 12658ab315ca4652baab3a5f69d785e3 (127.23.242.60:41799)
I20260806 00:58:24.777948 32160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:24.777857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.0.0.1:58098)
0806 00:58:24.777891 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:24.777920 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260806 00:58:24.778124 32159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: false } }
I20260806 00:58:24.778276 32159 leader_election.cc:288] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [CANDIDATE]: Term 2 election: Requested vote from peers ee13c8d77d44483c8dec48001f37be48 (127.23.242.62:38515), a6ca6242d6744e37a9d2b1aa76faef3c (127.23.242.61:41127)
I20260806 00:58:24.778396 32160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:24.778209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.0.0.1:58102)
0806 00:58:24.778322 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:24.778363 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260806 00:58:24.778404 31633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "12658ab315ca4652baab3a5f69d785e3"
I20260806 00:58:24.778461 31442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12658ab315ca4652baab3a5f69d785e3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c"
I20260806 00:58:24.778497 31633 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a6ca6242d6744e37a9d2b1aa76faef3c in current term 2: Already voted for candidate 12658ab315ca4652baab3a5f69d785e3 in this term.
W20260806 00:58:24.778642 31419 leader_election.cc:334] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ee13c8d77d44483c8dec48001f37be48 (127.23.242.62:38515): Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
I20260806 00:58:24.778692 31419 leader_election.cc:302] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a6ca6242d6744e37a9d2b1aa76faef3c; no voters: 12658ab315ca4652baab3a5f69d785e3, ee13c8d77d44483c8dec48001f37be48
I20260806 00:58:24.778541 31442 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 12658ab315ca4652baab3a5f69d785e3 in current term 2: Already voted for candidate a6ca6242d6744e37a9d2b1aa76faef3c in this term.
I20260806 00:58:24.778815 32158 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 00:58:24.778877 32163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:24.778334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.0.0.1:58106)
0806 00:58:24.778376 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:24.778399 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260806 00:58:24.779253 32163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:24.779163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.0.0.1:58122)
0806 00:58:24.779205 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:24.779231 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260806 00:58:24.779405 31609 leader_election.cc:334] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ee13c8d77d44483c8dec48001f37be48 (127.23.242.62:38515): Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
I20260806 00:58:24.779465 31609 leader_election.cc:302] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 12658ab315ca4652baab3a5f69d785e3; no voters: a6ca6242d6744e37a9d2b1aa76faef3c, ee13c8d77d44483c8dec48001f37be48
I20260806 00:58:24.779548 32159 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 00:58:25.269832 32168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:25.269477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.23.242.2:37901)
0806 00:58:25.269713 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:25.269773 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":78,"threads_started":1}
I20260806 00:58:25.270016 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:25.269813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.23.242.3:49299)
0806 00:58:25.269929 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:25.269965 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
I20260806 00:58:25.274405 32170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:25.274139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.23.242.1:37899)
0806 00:58:25.274288 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:25.274327 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":62,"threads_started":1}
I20260806 00:58:26.270637 32171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.270351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.23.242.2:42957)
0806 00:58:26.270491 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:26.270549 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":70,"threads_started":1}
I20260806 00:58:26.270733 32172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.270452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.23.242.3:32769)
0806 00:58:26.270613 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:26.270648 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":59,"threads_started":1}
I20260806 00:58:26.275429 32173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.275182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.23.242.1:60217)
0806 00:58:26.275348 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:26.275388 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":65,"threads_started":1}
I20260806 00:58:26.393011 32174 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 00:58:26.393138 32174 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: false } }
I20260806 00:58:26.393361 32174 leader_election.cc:288] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee13c8d77d44483c8dec48001f37be48 (127.23.242.62:38515), a6ca6242d6744e37a9d2b1aa76faef3c (127.23.242.61:41127)
I20260806 00:58:26.393695 31442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12658ab315ca4652baab3a5f69d785e3" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" is_pre_election: true
I20260806 00:58:26.393801 31442 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12658ab315ca4652baab3a5f69d785e3 in term 2.
I20260806 00:58:26.394042 31608 leader_election.cc:302] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12658ab315ca4652baab3a5f69d785e3, a6ca6242d6744e37a9d2b1aa76faef3c; no voters:
I20260806 00:58:26.394117 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.393694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.0.0.1:58306)
0806 00:58:26.394015 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:26.394072 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":133,"threads_started":1}
I20260806 00:58:26.394150 32174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 00:58:26.394232 32174 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 00:58:26.394265 32174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 2 FOLLOWER]: Advancing to term 3
I20260806 00:58:26.394877 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.394764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.0.0.1:58328)
0806 00:58:26.394825 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:26.394856 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260806 00:58:26.394960 31609 leader_election.cc:334] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ee13c8d77d44483c8dec48001f37be48 (127.23.242.62:38515): Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
I20260806 00:58:26.395308 32174 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: false } }
I20260806 00:58:26.395483 32174 leader_election.cc:288] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [CANDIDATE]: Term 3 election: Requested vote from peers ee13c8d77d44483c8dec48001f37be48 (127.23.242.62:38515), a6ca6242d6744e37a9d2b1aa76faef3c (127.23.242.61:41127)
I20260806 00:58:26.395689 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.395561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.0.0.1:58332)
0806 00:58:26.395629 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:26.395658 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 00:58:26.395723 31442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12658ab315ca4652baab3a5f69d785e3" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c"
I20260806 00:58:26.395828 31442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 2 FOLLOWER]: Advancing to term 3
I20260806 00:58:26.396528 31442 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12658ab315ca4652baab3a5f69d785e3 in term 3.
I20260806 00:58:26.396747 31608 leader_election.cc:302] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12658ab315ca4652baab3a5f69d785e3, a6ca6242d6744e37a9d2b1aa76faef3c; no voters:
I20260806 00:58:26.396893 32174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 00:58:26.396903 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.396772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.0.0.1:58346)
0806 00:58:26.396829 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:26.396868 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260806 00:58:26.397024 31609 leader_election.cc:334] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ee13c8d77d44483c8dec48001f37be48 (127.23.242.62:38515): Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
I20260806 00:58:26.397048 32174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [term 3 LEADER]: Becoming Leader. State: Replica: 12658ab315ca4652baab3a5f69d785e3, State: Running, Role: LEADER
I20260806 00:58:26.397118 32174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: false } }
I20260806 00:58:26.397511 32177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12658ab315ca4652baab3a5f69d785e3. Latest consensus state: current_term: 3 leader_uuid: "12658ab315ca4652baab3a5f69d785e3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: false } } }
I20260806 00:58:26.397588 32177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:26.397773 32179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 00:58:26.398221 32179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 00:58:26.398327 32179 catalog_manager.cc:1259] Loaded cluster ID: 3b0f30832a7d4834950ee317a3e241e2
I20260806 00:58:26.398365 32179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 00:58:26.398563 32179 catalog_manager.cc:1504] Loading token signing keys...
I20260806 00:58:26.398706 32179 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3: Loaded TSK: 0
I20260806 00:58:26.398860 32179 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 00:58:26.483613 31623 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 00:58:26.485283 31623 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 00:58:26.486636 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.486473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.0.0.1:58372)
0806 00:58:26.486513 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:26.486552 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260806 00:58:26.486685 31442 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [term 3 FOLLOWER]: Refusing update from remote peer 12658ab315ca4652baab3a5f69d785e3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260806 00:58:26.486990 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.486879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.0.0.1:58376)
0806 00:58:26.486936 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:26.486965 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 00:58:26.487071 31609 consensus_peers.cc:487] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 -> Peer ee13c8d77d44483c8dec48001f37be48 (127.23.242.62:38515): Couldn't send request to peer ee13c8d77d44483c8dec48001f37be48. Status: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 00:58:26.487216 32174 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260806 00:58:26.488282 32181 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12658ab315ca4652baab3a5f69d785e3. Latest consensus state: current_term: 3 leader_uuid: "12658ab315ca4652baab3a5f69d785e3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: false } } }
I20260806 00:58:26.488404 32181 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 00:58:26.488832 32180 mvcc.cc:205] Tried to move back new op lower bound from 7315365504965988352 to 7315365504603017216. Current Snapshot: MvccSnapshot[applied={T|T < 7315365504965988352}]
I20260806 00:58:26.488958 32177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "12658ab315ca4652baab3a5f69d785e3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: false } } }
I20260806 00:58:26.489048 32177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:26.489156 32181 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "12658ab315ca4652baab3a5f69d785e3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee13c8d77d44483c8dec48001f37be48" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38515 } } peers { permanent_uuid: "a6ca6242d6744e37a9d2b1aa76faef3c" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 41127 } attrs { promote: false } } peers { permanent_uuid: "12658ab315ca4652baab3a5f69d785e3" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 41799 } attrs { promote: false } } }
I20260806 00:58:26.489228 32181 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ca6242d6744e37a9d2b1aa76faef3c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 00:58:26.489346 32182 mvcc.cc:205] Tried to move back new op lower bound from 7315365504965988352 to 7315365504603017216. Current Snapshot: MvccSnapshot[applied={T|T < 7315365504965988352 or (T in {7315365504965988352})}]
I20260806 00:58:26.495080 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.491164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:39943 (local address 127.0.0.1:36536)
0806 00:58:26.491223 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:26.491236 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:26.491298 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:26.491826 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:26.491828 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:26.491830 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:26.491894 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:26.491896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:26.492762 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:26.492763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:26.494047 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:26.494049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:26.494845 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:26.494846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:26.494913 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:26.494918 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 00:58:26.494948 (+ 30us) client_negotiation.cc:240] Negotiation successful
0806 00:58:26.494989 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260806 00:58:26.495427 32188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.491379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36536 (local address 127.23.242.3:39943)
0806 00:58:26.491509 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:26.491512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:26.491659 (+ 147us) server_negotiation.cc:407] Connection header received
0806 00:58:26.491694 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:26.491695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:26.491709 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:26.491983 (+ 274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:26.492024 (+ 41us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:26.492665 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:26.494133 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:26.494744 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:26.495270 (+ 526us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:26.495289 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:26.495310 (+ 21us) server_negotiation.cc:299] Negotiation successful
0806 00:58:26.495357 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260806 00:58:26.495618 32189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.491164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:45035 (local address 127.0.0.1:36276)
0806 00:58:26.491742 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:26.491757 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:26.491826 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:26.492313 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:26.492314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:26.492316 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:26.492354 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:26.492356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:26.493542 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:26.493542 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:26.494541 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:26.494543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:26.495360 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:26.495361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:26.495426 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:26.495429 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 00:58:26.495479 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 00:58:26.495529 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":25,"thread_start_us":142,"threads_started":1}
I20260806 00:58:26.495870 32187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.491355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36276 (local address 127.23.242.2:45035)
0806 00:58:26.491484 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:26.491486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:26.491892 (+ 406us) server_negotiation.cc:407] Connection header received
0806 00:58:26.491925 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:26.491927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:26.491941 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:26.492233 (+ 292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:26.492602 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:26.493461 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:26.494656 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:26.495256 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:26.495319 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:26.495330 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:26.495761 (+ 431us) server_negotiation.cc:299] Negotiation successful
0806 00:58:26.495812 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":86,"threads_started":1}
I20260806 00:58:26.495872 32021 tablet_service.cc:1467] Processing CreateTablet for tablet 7b2fc477cbff4b6686b67dc6d8eee6fd (DEFAULT_TABLE table=test-table [id=7054aaae78d94fd2ad28feb57f90207b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:26.496037 32021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b2fc477cbff4b6686b67dc6d8eee6fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:26.496106 32186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.491446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:42435 (local address 127.0.0.1:56930)
0806 00:58:26.492789 (+ 1343us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:26.492799 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:26.492858 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:26.493026 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:26.493027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:26.493029 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:26.493080 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:26.493082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:26.493960 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:26.493961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:26.494932 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:26.494933 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:26.495827 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:26.495829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:26.495897 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:26.495902 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 00:58:26.495973 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 00:58:26.496024 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1320,"thread_start_us":211,"threads_started":1}
I20260806 00:58:26.496371 32190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.491774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56930 (local address 127.23.242.1:42435)
0806 00:58:26.491908 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:26.491910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:26.492881 (+ 971us) server_negotiation.cc:407] Connection header received
0806 00:58:26.492914 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:26.492915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:26.492926 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:26.493017 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:26.493139 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:26.493874 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:26.495022 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:26.495594 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:26.495750 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:26.495789 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:26.496255 (+ 466us) server_negotiation.cc:299] Negotiation successful
0806 00:58:26.496306 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260806 00:58:26.496730 31738 tablet_service.cc:1467] Processing CreateTablet for tablet 7b2fc477cbff4b6686b67dc6d8eee6fd (DEFAULT_TABLE table=test-table [id=7054aaae78d94fd2ad28feb57f90207b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:26.496868 31738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b2fc477cbff4b6686b67dc6d8eee6fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:26.497737 32191 tablet_bootstrap.cc:492] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ec07cecb7fe24fed939834282411206d: Bootstrap starting.
I20260806 00:58:26.497833 31881 tablet_service.cc:1467] Processing CreateTablet for tablet 7b2fc477cbff4b6686b67dc6d8eee6fd (DEFAULT_TABLE table=test-table [id=7054aaae78d94fd2ad28feb57f90207b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:26.498179 31881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b2fc477cbff4b6686b67dc6d8eee6fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:26.498412 32191 tablet_bootstrap.cc:654] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ec07cecb7fe24fed939834282411206d: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:26.498394 32192 tablet_bootstrap.cc:492] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ba9ec1e1be664d3b9e705abaf8f50594: Bootstrap starting.
I20260806 00:58:26.499022 32192 tablet_bootstrap.cc:654] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ba9ec1e1be664d3b9e705abaf8f50594: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:26.499737 32195 tablet_bootstrap.cc:492] T 7b2fc477cbff4b6686b67dc6d8eee6fd P 4765fb0b489240bb920242b63a57e71f: Bootstrap starting.
I20260806 00:58:26.500174 32192 tablet_bootstrap.cc:492] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ba9ec1e1be664d3b9e705abaf8f50594: No bootstrap required, opened a new log
I20260806 00:58:26.500331 32192 ts_tablet_manager.cc:1397] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ba9ec1e1be664d3b9e705abaf8f50594: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260806 00:58:26.500409 32191 tablet_bootstrap.cc:492] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ec07cecb7fe24fed939834282411206d: No bootstrap required, opened a new log
I20260806 00:58:26.500463 32191 ts_tablet_manager.cc:1397] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ec07cecb7fe24fed939834282411206d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260806 00:58:26.500620 32191 raft_consensus.cc:348] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ec07cecb7fe24fed939834282411206d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07cecb7fe24fed939834282411206d" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39943 } } peers { permanent_uuid: "4765fb0b489240bb920242b63a57e71f" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45035 } } peers { permanent_uuid: "ba9ec1e1be664d3b9e705abaf8f50594" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42435 } }
I20260806 00:58:26.500707 32191 raft_consensus.cc:374] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ec07cecb7fe24fed939834282411206d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:26.500736 32191 raft_consensus.cc:729] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ec07cecb7fe24fed939834282411206d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec07cecb7fe24fed939834282411206d, State: Initialized, Role: FOLLOWER
I20260806 00:58:26.500666 32192 raft_consensus.cc:348] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ba9ec1e1be664d3b9e705abaf8f50594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07cecb7fe24fed939834282411206d" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39943 } } peers { permanent_uuid: "4765fb0b489240bb920242b63a57e71f" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45035 } } peers { permanent_uuid: "ba9ec1e1be664d3b9e705abaf8f50594" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42435 } }
I20260806 00:58:26.500792 32191 consensus_queue.cc:260] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ec07cecb7fe24fed939834282411206d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07cecb7fe24fed939834282411206d" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39943 } } peers { permanent_uuid: "4765fb0b489240bb920242b63a57e71f" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45035 } } peers { permanent_uuid: "ba9ec1e1be664d3b9e705abaf8f50594" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42435 } }
I20260806 00:58:26.500829 32192 raft_consensus.cc:374] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ba9ec1e1be664d3b9e705abaf8f50594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:26.500860 32192 raft_consensus.cc:729] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ba9ec1e1be664d3b9e705abaf8f50594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba9ec1e1be664d3b9e705abaf8f50594, State: Initialized, Role: FOLLOWER
I20260806 00:58:26.500994 32191 ts_tablet_manager.cc:1428] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ec07cecb7fe24fed939834282411206d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260806 00:58:26.500211 32195 tablet_bootstrap.cc:654] T 7b2fc477cbff4b6686b67dc6d8eee6fd P 4765fb0b489240bb920242b63a57e71f: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:26.501338 32172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.501232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.23.242.3:40431)
0806 00:58:26.501274 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:26.501306 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260806 00:58:26.501617 32192 consensus_queue.cc:260] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ba9ec1e1be664d3b9e705abaf8f50594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07cecb7fe24fed939834282411206d" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39943 } } peers { permanent_uuid: "4765fb0b489240bb920242b63a57e71f" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45035 } } peers { permanent_uuid: "ba9ec1e1be664d3b9e705abaf8f50594" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42435 } }
I20260806 00:58:26.501876 32088 heartbeater.cc:503] Master 127.23.242.60:41799 was elected leader, sending a full tablet report...
I20260806 00:58:26.501964 32192 ts_tablet_manager.cc:1428] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ba9ec1e1be664d3b9e705abaf8f50594: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260806 00:58:26.502067 32195 tablet_bootstrap.cc:492] T 7b2fc477cbff4b6686b67dc6d8eee6fd P 4765fb0b489240bb920242b63a57e71f: No bootstrap required, opened a new log
I20260806 00:58:26.502146 32195 ts_tablet_manager.cc:1397] T 7b2fc477cbff4b6686b67dc6d8eee6fd P 4765fb0b489240bb920242b63a57e71f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 00:58:26.502380 32173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.502221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.23.242.1:59585)
0806 00:58:26.502319 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:26.502348 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260806 00:58:26.502354 32195 raft_consensus.cc:348] T 7b2fc477cbff4b6686b67dc6d8eee6fd P 4765fb0b489240bb920242b63a57e71f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07cecb7fe24fed939834282411206d" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39943 } } peers { permanent_uuid: "4765fb0b489240bb920242b63a57e71f" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45035 } } peers { permanent_uuid: "ba9ec1e1be664d3b9e705abaf8f50594" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42435 } }
I20260806 00:58:26.502405 31808 heartbeater.cc:503] Master 127.23.242.60:41799 was elected leader, sending a full tablet report...
I20260806 00:58:26.502431 32195 raft_consensus.cc:374] T 7b2fc477cbff4b6686b67dc6d8eee6fd P 4765fb0b489240bb920242b63a57e71f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:26.502453 32195 raft_consensus.cc:729] T 7b2fc477cbff4b6686b67dc6d8eee6fd P 4765fb0b489240bb920242b63a57e71f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4765fb0b489240bb920242b63a57e71f, State: Initialized, Role: FOLLOWER
I20260806 00:58:26.502511 32195 consensus_queue.cc:260] T 7b2fc477cbff4b6686b67dc6d8eee6fd P 4765fb0b489240bb920242b63a57e71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07cecb7fe24fed939834282411206d" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39943 } } peers { permanent_uuid: "4765fb0b489240bb920242b63a57e71f" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45035 } } peers { permanent_uuid: "ba9ec1e1be664d3b9e705abaf8f50594" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42435 } }
I20260806 00:58:26.502761 32195 ts_tablet_manager.cc:1428] T 7b2fc477cbff4b6686b67dc6d8eee6fd P 4765fb0b489240bb920242b63a57e71f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 00:58:26.502812 32173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.502723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.23.242.1:41011)
0806 00:58:26.502762 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:26.502784 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260806 00:58:26.503259 32171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.503167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.23.242.2:51975)
0806 00:58:26.503209 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:26.503233 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260806 00:58:26.503551 31948 heartbeater.cc:503] Master 127.23.242.60:41799 was elected leader, sending a full tablet report...
I20260806 00:58:26.503695 32171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.503597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.23.242.2:59473)
0806 00:58:26.503647 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:26.503670 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260806 00:58:26.538750 32198 raft_consensus.cc:482] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ba9ec1e1be664d3b9e705abaf8f50594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 00:58:26.538863 32198 raft_consensus.cc:504] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ba9ec1e1be664d3b9e705abaf8f50594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07cecb7fe24fed939834282411206d" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39943 } } peers { permanent_uuid: "4765fb0b489240bb920242b63a57e71f" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45035 } } peers { permanent_uuid: "ba9ec1e1be664d3b9e705abaf8f50594" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42435 } }
I20260806 00:58:26.539583 32198 leader_election.cc:288] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ba9ec1e1be664d3b9e705abaf8f50594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec07cecb7fe24fed939834282411206d (127.23.242.3:39943), 4765fb0b489240bb920242b63a57e71f (127.23.242.2:45035)
I20260806 00:58:26.544579 32187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.539642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:37081 (local address 127.23.242.2:45035)
0806 00:58:26.539772 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:26.539775 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:26.540497 (+ 722us) server_negotiation.cc:407] Connection header received
0806 00:58:26.540528 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:26.540530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:26.540541 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:26.540598 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:26.540777 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:26.541583 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:26.543430 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:26.544069 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:26.544191 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:26.544204 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:26.544476 (+ 272us) server_negotiation.cc:299] Negotiation successful
0806 00:58:26.544519 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 00:58:26.544653 32200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.539553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:45035 (local address 127.23.242.1:37081)
0806 00:58:26.540325 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:26.540340 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:26.540430 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:26.540634 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:26.540635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:26.540638 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:26.540700 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:26.540701 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:26.541808 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:26.541810 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:26.543313 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:26.543316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:26.544210 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:26.544211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:26.544317 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:26.544323 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 00:58:26.544454 (+ 131us) client_negotiation.cc:240] Negotiation successful
0806 00:58:26.544513 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":268,"thread_start_us":303,"threads_started":1}
I20260806 00:58:26.544826 32173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.539553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:39943 (local address 127.23.242.1:46293)
0806 00:58:26.539633 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:26.539647 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:26.539920 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:26.540241 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:26.540242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:26.540246 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:26.540378 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:26.540381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:26.541650 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:26.541651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:26.543326 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:26.543328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:26.544449 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:26.544450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:26.544647 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:26.544653 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 00:58:26.544714 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 00:58:26.544756 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":2}
I20260806 00:58:26.545010 32188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.539707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:46293 (local address 127.23.242.3:39943)
0806 00:58:26.539763 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:26.539765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:26.539782 (+ 17us) server_negotiation.cc:407] Connection header received
0806 00:58:26.540097 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:26.540099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:26.540111 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:26.540181 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:26.540487 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:26.541326 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:26.543563 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:26.544201 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:26.544274 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:26.544286 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:26.544904 (+ 618us) server_negotiation.cc:299] Negotiation successful
0806 00:58:26.544963 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 00:58:26.545282 32041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b2fc477cbff4b6686b67dc6d8eee6fd" candidate_uuid: "ba9ec1e1be664d3b9e705abaf8f50594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec07cecb7fe24fed939834282411206d" is_pre_election: true
I20260806 00:58:26.545324 31901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b2fc477cbff4b6686b67dc6d8eee6fd" candidate_uuid: "ba9ec1e1be664d3b9e705abaf8f50594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4765fb0b489240bb920242b63a57e71f" is_pre_election: true
I20260806 00:58:26.545430 31901 raft_consensus.cc:2454] T 7b2fc477cbff4b6686b67dc6d8eee6fd P 4765fb0b489240bb920242b63a57e71f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba9ec1e1be664d3b9e705abaf8f50594 in term 0.
I20260806 00:58:26.545445 32041 raft_consensus.cc:2454] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ec07cecb7fe24fed939834282411206d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba9ec1e1be664d3b9e705abaf8f50594 in term 0.
I20260806 00:58:26.545658 31691 leader_election.cc:302] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ba9ec1e1be664d3b9e705abaf8f50594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4765fb0b489240bb920242b63a57e71f, ba9ec1e1be664d3b9e705abaf8f50594; no voters:
I20260806 00:58:26.545816 32198 raft_consensus.cc:2781] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ba9ec1e1be664d3b9e705abaf8f50594 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 00:58:26.545881 32198 raft_consensus.cc:482] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ba9ec1e1be664d3b9e705abaf8f50594 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 00:58:26.545897 32198 raft_consensus.cc:3037] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ba9ec1e1be664d3b9e705abaf8f50594 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:26.546765 32198 raft_consensus.cc:504] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ba9ec1e1be664d3b9e705abaf8f50594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07cecb7fe24fed939834282411206d" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39943 } } peers { permanent_uuid: "4765fb0b489240bb920242b63a57e71f" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45035 } } peers { permanent_uuid: "ba9ec1e1be664d3b9e705abaf8f50594" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42435 } }
I20260806 00:58:26.546988 32198 leader_election.cc:288] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ba9ec1e1be664d3b9e705abaf8f50594 [CANDIDATE]: Term 1 election: Requested vote from peers ec07cecb7fe24fed939834282411206d (127.23.242.3:39943), 4765fb0b489240bb920242b63a57e71f (127.23.242.2:45035)
I20260806 00:58:26.547075 31901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b2fc477cbff4b6686b67dc6d8eee6fd" candidate_uuid: "ba9ec1e1be664d3b9e705abaf8f50594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4765fb0b489240bb920242b63a57e71f"
I20260806 00:58:26.547079 32041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b2fc477cbff4b6686b67dc6d8eee6fd" candidate_uuid: "ba9ec1e1be664d3b9e705abaf8f50594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec07cecb7fe24fed939834282411206d"
I20260806 00:58:26.547144 31901 raft_consensus.cc:3037] T 7b2fc477cbff4b6686b67dc6d8eee6fd P 4765fb0b489240bb920242b63a57e71f [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:26.547156 32041 raft_consensus.cc:3037] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ec07cecb7fe24fed939834282411206d [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:26.547696 31901 raft_consensus.cc:2454] T 7b2fc477cbff4b6686b67dc6d8eee6fd P 4765fb0b489240bb920242b63a57e71f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba9ec1e1be664d3b9e705abaf8f50594 in term 1.
I20260806 00:58:26.547881 31691 leader_election.cc:302] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ba9ec1e1be664d3b9e705abaf8f50594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4765fb0b489240bb920242b63a57e71f, ba9ec1e1be664d3b9e705abaf8f50594; no voters:
I20260806 00:58:26.548005 32041 raft_consensus.cc:2454] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ec07cecb7fe24fed939834282411206d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba9ec1e1be664d3b9e705abaf8f50594 in term 1.
I20260806 00:58:26.548014 32198 raft_consensus.cc:2781] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ba9ec1e1be664d3b9e705abaf8f50594 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:26.548233 32198 raft_consensus.cc:686] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ba9ec1e1be664d3b9e705abaf8f50594 [term 1 LEADER]: Becoming Leader. State: Replica: ba9ec1e1be664d3b9e705abaf8f50594, State: Running, Role: LEADER
I20260806 00:58:26.548450 32198 consensus_queue.cc:237] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ba9ec1e1be664d3b9e705abaf8f50594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07cecb7fe24fed939834282411206d" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39943 } } peers { permanent_uuid: "4765fb0b489240bb920242b63a57e71f" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45035 } } peers { permanent_uuid: "ba9ec1e1be664d3b9e705abaf8f50594" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42435 } }
I20260806 00:58:26.548588 32173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.548461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.23.242.1:49861)
0806 00:58:26.548520 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:26.548561 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260806 00:58:26.549337 31622 catalog_manager.cc:5495] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ba9ec1e1be664d3b9e705abaf8f50594 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba9ec1e1be664d3b9e705abaf8f50594 (127.23.242.1). New cstate: current_term: 1 leader_uuid: "ba9ec1e1be664d3b9e705abaf8f50594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec07cecb7fe24fed939834282411206d" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4765fb0b489240bb920242b63a57e71f" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba9ec1e1be664d3b9e705abaf8f50594" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 42435 } health_report { overall_health: HEALTHY } } }
I20260806 00:58:26.556478 32190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.553944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56936 (local address 127.23.242.1:42435)
0806 00:58:26.553997 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:26.553999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:26.554008 (+ 9us) server_negotiation.cc:407] Connection header received
0806 00:58:26.554098 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:26.554099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:26.554119 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:26.554201 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 00:58:26.554301 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:26.554872 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:26.555631 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:26.555861 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:26.555928 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:26.556115 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 00:58:26.556248 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 00:58:26.556310 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:26.556398 (+ 88us) server_negotiation.cc:299] Negotiation successful
0806 00:58:26.556419 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 00:58:26.557984 31901 raft_consensus.cc:1261] T 7b2fc477cbff4b6686b67dc6d8eee6fd P 4765fb0b489240bb920242b63a57e71f [term 1 FOLLOWER]: Refusing update from remote peer ba9ec1e1be664d3b9e705abaf8f50594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:26.558338 32201 consensus_queue.cc:1035] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ba9ec1e1be664d3b9e705abaf8f50594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4765fb0b489240bb920242b63a57e71f" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:26.558451 32041 raft_consensus.cc:1261] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ec07cecb7fe24fed939834282411206d [term 1 FOLLOWER]: Refusing update from remote peer ba9ec1e1be664d3b9e705abaf8f50594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:26.558802 32201 consensus_queue.cc:1035] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ba9ec1e1be664d3b9e705abaf8f50594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec07cecb7fe24fed939834282411206d" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 39943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:26.559229 32171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.559107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.23.242.2:38009)
0806 00:58:26.559154 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:26.559188 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260806 00:58:26.559649 32172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.559553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.23.242.3:37601)
0806 00:58:26.559593 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:26.559620 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260806 00:58:26.561506 32171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.561365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.23.242.2:33483)
0806 00:58:26.561428 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:26.561461 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 00:58:26.562018 32173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.561830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.23.242.1:60529)
0806 00:58:26.561934 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:26.561966 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260806 00:58:26.562167 32172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.562069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38515 (local address 127.23.242.3:51669)
0806 00:58:26.562112 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:26.562137 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:38515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260806 00:58:26.564245 31623 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:58:26.564358 31623 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:58:26.565794 31623 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 12658ab315ca4652baab3a5f69d785e3: Sending DeleteTablet for 3 replicas of tablet 7b2fc477cbff4b6686b67dc6d8eee6fd
I20260806 00:58:26.566121 32021 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b2fc477cbff4b6686b67dc6d8eee6fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:26 UTC) from {username='oryx', principal='oryx/127.23.242.60@KRBTEST.COM'} at 127.0.0.1:36536
I20260806 00:58:26.566504 31881 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b2fc477cbff4b6686b67dc6d8eee6fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:26 UTC) from {username='oryx', principal='oryx/127.23.242.60@KRBTEST.COM'} at 127.0.0.1:36276
I20260806 00:58:26.566777 31738 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b2fc477cbff4b6686b67dc6d8eee6fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:26 UTC) from {username='oryx', principal='oryx/127.23.242.60@KRBTEST.COM'} at 127.0.0.1:56930
I20260806 00:58:26.566846 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 31583
I20260806 00:58:26.570792 32215 tablet_replica.cc:330] T 7b2fc477cbff4b6686b67dc6d8eee6fd P 4765fb0b489240bb920242b63a57e71f: stopping tablet replica
I20260806 00:58:26.570891 32215 raft_consensus.cc:2229] T 7b2fc477cbff4b6686b67dc6d8eee6fd P 4765fb0b489240bb920242b63a57e71f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:26.570967 32215 raft_consensus.cc:2258] T 7b2fc477cbff4b6686b67dc6d8eee6fd P 4765fb0b489240bb920242b63a57e71f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:26.571553 32215 ts_tablet_manager.cc:1905] T 7b2fc477cbff4b6686b67dc6d8eee6fd P 4765fb0b489240bb920242b63a57e71f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:26.573591 32215 ts_tablet_manager.cc:1918] T 7b2fc477cbff4b6686b67dc6d8eee6fd P 4765fb0b489240bb920242b63a57e71f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 00:58:26.573812 32215 log.cc:1196] T 7b2fc477cbff4b6686b67dc6d8eee6fd P 4765fb0b489240bb920242b63a57e71f: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-1/wal/wals/7b2fc477cbff4b6686b67dc6d8eee6fd
I20260806 00:58:26.574080 32215 ts_tablet_manager.cc:1939] T 7b2fc477cbff4b6686b67dc6d8eee6fd P 4765fb0b489240bb920242b63a57e71f: Deleting consensus metadata
I20260806 00:58:26.574479 31606 catalog_manager.cc:4841] TS 4765fb0b489240bb920242b63a57e71f (127.23.242.2:45035): tablet 7b2fc477cbff4b6686b67dc6d8eee6fd (table test-table [id=7054aaae78d94fd2ad28feb57f90207b]) successfully deleted
I20260806 00:58:26.576045 32186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:26.575905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:42435 (local address 127.0.0.1:56942)
0806 00:58:26.575970 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:26.576004 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.1:42435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260806 00:58:26.576224 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 31816
W20260806 00:58:26.576227 31606 catalog_manager.cc:4584] TS ba9ec1e1be664d3b9e705abaf8f50594 (127.23.242.1:42435): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7b2fc477cbff4b6686b67dc6d8eee6fd: Network error: Client connection negotiation failed: client connection to 127.23.242.1:42435: connect: Connection refused (error 111)
I20260806 00:58:26.578985 32214 tablet_replica.cc:330] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ec07cecb7fe24fed939834282411206d: stopping tablet replica
I20260806 00:58:26.579092 32214 raft_consensus.cc:2229] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ec07cecb7fe24fed939834282411206d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:26.579160 32214 raft_consensus.cc:2258] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ec07cecb7fe24fed939834282411206d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:26.579587 32214 ts_tablet_manager.cc:1905] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ec07cecb7fe24fed939834282411206d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:26.581270 32214 ts_tablet_manager.cc:1918] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ec07cecb7fe24fed939834282411206d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 00:58:26.581354 32214 log.cc:1196] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ec07cecb7fe24fed939834282411206d: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785977869987768-24520-0/minicluster-data/ts-2/wal/wals/7b2fc477cbff4b6686b67dc6d8eee6fd
I20260806 00:58:26.581612 32214 ts_tablet_manager.cc:1939] T 7b2fc477cbff4b6686b67dc6d8eee6fd P ec07cecb7fe24fed939834282411206d: Deleting consensus metadata
I20260806 00:58:26.582015 31607 catalog_manager.cc:4841] TS ec07cecb7fe24fed939834282411206d (127.23.242.3:39943): tablet 7b2fc477cbff4b6686b67dc6d8eee6fd (table test-table [id=7054aaae78d94fd2ad28feb57f90207b]) successfully deleted
I20260806 00:58:26.583046 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 31955
I20260806 00:58:26.590751 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 31296
I20260806 00:58:26.597962 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 31487
2026-08-06T00:58:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6773 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 00:58:26 dist-test-slave-dhph krb5kdc[32219](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 00:58:26 dist-test-slave-dhph krb5kdc[32219](info): set up 2 sockets
Aug 06 00:58:26 dist-test-slave-dhph krb5kdc[32219](info): commencing operation
krb5kdc: starting...
W20260806 00:58:27.757686 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 00:58:27 dist-test-slave-dhph krb5kdc[32219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T00:58:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T00:58:27Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.62.keytab.
Entry for principal oryx/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.62.keytab.
I20260806 00:58:27.903013 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:39515
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.62.keytab
--principal=oryx/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:39515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:28.003605 32232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:28.003947 32232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:28.003979 32232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:28.006767 32232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 00:58:28.006860 32232 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 00:58:28.006878 32232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:28.006896 32232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:28.006911 32232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 00:58:28.006927 32232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 00:58:28.009095 32232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:39515
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.62.keytab
--principal=oryx/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:39515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:28.009423 32232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:28.009706 32232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:28.016357 32237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:28.016343 32238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:28.016342 32240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:28.016412 32232 server_base.cc:1034] running on GCE node
I20260806 00:58:28.016873 32232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:28.017269 32232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:28.018556 32232 hybrid_clock.cc:648] HybridClock initialized: now 1785977908018515 us; error 51 us; skew 500 ppm
Aug 06 00:58:28 dist-test-slave-dhph krb5kdc[32219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977908, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:28.024328 32232 init.cc:377] Logged in from keytab as oryx/127.23.242.62@KRBTEST.COM (short username oryx)
I20260806 00:58:28.024968 32232 webserver.cc:460] Webserver started at http://127.23.242.62:42117/ using document root <none> and password file <none>
I20260806 00:58:28.025259 32232 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:28.025342 32232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:28.025518 32232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:28.026957 32232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "d6398079945c47f6bc6b7163ae969179"
format_stamp: "Formatted at 2026-08-06 00:58:28 on dist-test-slave-dhph"
server_key: "972be76b412f8731c1b398e39331d100"
server_key_iv: "81df1194277cb84c2383c63203ad735a"
server_key_version: "encryptionkey@0"
I20260806 00:58:28.027366 32232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "d6398079945c47f6bc6b7163ae969179"
format_stamp: "Formatted at 2026-08-06 00:58:28 on dist-test-slave-dhph"
server_key: "972be76b412f8731c1b398e39331d100"
server_key_iv: "81df1194277cb84c2383c63203ad735a"
server_key_version: "encryptionkey@0"
I20260806 00:58:28.029094 32232 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 00:58:28.030174 32248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:28.031589 32232 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 00:58:28.031818 32232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "d6398079945c47f6bc6b7163ae969179"
format_stamp: "Formatted at 2026-08-06 00:58:28 on dist-test-slave-dhph"
server_key: "972be76b412f8731c1b398e39331d100"
server_key_iv: "81df1194277cb84c2383c63203ad735a"
server_key_version: "encryptionkey@0"
I20260806 00:58:28.031950 32232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:28.044301 32232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:28.047435 32232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:28.047631 32232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:28.053556 32232 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:39515
I20260806 00:58:28.053613 32311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:39515 every 8 connection(s)
I20260806 00:58:28.053997 32232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 00:58:28.054839 32312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:28.057541 32312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6398079945c47f6bc6b7163ae969179: Bootstrap starting.
I20260806 00:58:28.058215 32312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6398079945c47f6bc6b7163ae969179: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:28.058486 32312 log.cc:824] T 00000000000000000000000000000000 P d6398079945c47f6bc6b7163ae969179: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:28.059301 32312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6398079945c47f6bc6b7163ae969179: No bootstrap required, opened a new log
I20260806 00:58:28.060618 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 32232
I20260806 00:58:28.060778 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 00:58:28.060992 24520 external_mini_cluster.cc:1428] Setting key bd01cd416b05ad1beb99b2c9b91bfb2a
I20260806 00:58:28.061873 32312 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6398079945c47f6bc6b7163ae969179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6398079945c47f6bc6b7163ae969179" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39515 } }
I20260806 00:58:28.062062 32312 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6398079945c47f6bc6b7163ae969179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:28.062115 32312 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6398079945c47f6bc6b7163ae969179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6398079945c47f6bc6b7163ae969179, State: Initialized, Role: FOLLOWER
I20260806 00:58:28.062247 32312 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6398079945c47f6bc6b7163ae969179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6398079945c47f6bc6b7163ae969179" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39515 } }
I20260806 00:58:28.062348 32312 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6398079945c47f6bc6b7163ae969179 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 00:58:28.062408 32312 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6398079945c47f6bc6b7163ae969179 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 00:58:28.062460 32312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6398079945c47f6bc6b7163ae969179 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:28.063092 32312 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6398079945c47f6bc6b7163ae969179 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6398079945c47f6bc6b7163ae969179" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39515 } }
I20260806 00:58:28.063216 32312 leader_election.cc:302] T 00000000000000000000000000000000 P d6398079945c47f6bc6b7163ae969179 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6398079945c47f6bc6b7163ae969179; no voters:
I20260806 00:58:28.063414 32312 leader_election.cc:288] T 00000000000000000000000000000000 P d6398079945c47f6bc6b7163ae969179 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 00:58:28.063447 32317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6398079945c47f6bc6b7163ae969179 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:28.063584 32317 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6398079945c47f6bc6b7163ae969179 [term 1 LEADER]: Becoming Leader. State: Replica: d6398079945c47f6bc6b7163ae969179, State: Running, Role: LEADER
I20260806 00:58:28.064042 32312 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6398079945c47f6bc6b7163ae969179 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 00:58:28.064088 32317 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6398079945c47f6bc6b7163ae969179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6398079945c47f6bc6b7163ae969179" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39515 } }
I20260806 00:58:28.065053 32318 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6398079945c47f6bc6b7163ae969179 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6398079945c47f6bc6b7163ae969179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6398079945c47f6bc6b7163ae969179" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39515 } } }
I20260806 00:58:28.065189 32318 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6398079945c47f6bc6b7163ae969179 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:28.065689 32327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 00:58:28.065667 32319 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6398079945c47f6bc6b7163ae969179 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6398079945c47f6bc6b7163ae969179. Latest consensus state: current_term: 1 leader_uuid: "d6398079945c47f6bc6b7163ae969179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6398079945c47f6bc6b7163ae969179" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39515 } } }
I20260806 00:58:28.065809 32319 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6398079945c47f6bc6b7163ae969179 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:28.066461 32327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 00:58:28.068216 32327 catalog_manager.cc:1347] Generated new cluster ID: 84579fef6f74489fb210856c8a13e19f
I20260806 00:58:28.068274 32327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 00:58:28 dist-test-slave-dhph krb5kdc[32219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM
I20260806 00:58:28.073180 32316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.061727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34496 (local address 127.23.242.62:39515)
0806 00:58:28.061981 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:28.061990 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:28.062019 (+ 29us) server_negotiation.cc:407] Connection header received
0806 00:58:28.062179 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:28.062187 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:28.062263 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:28.062431 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:28.063222 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.064872 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.066229 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.066788 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.066934 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.069734 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:28.069739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:28.069739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:28.069759 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:28.071573 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:28.071967 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:28.071971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:28.071971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:28.072080 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:28.072294 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:28.072295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:28.072296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:28.072542 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:28.072730 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:28.072806 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 00:58:28.072951 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":104,"threads_started":1}
I20260806 00:58:28.078717 32327 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 00:58:28.079542 32327 catalog_manager.cc:1504] Loading token signing keys...
I20260806 00:58:28.091552 32327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6398079945c47f6bc6b7163ae969179: Generated new TSK 0
I20260806 00:58:28.091807 32327 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.242.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.242.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.242.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.1.keytab.
Entry for principal oryx/127.23.242.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.1.keytab.
I20260806 00:58:28.136137 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:0
--local_ip_for_outbound_sockets=127.23.242.1
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:39515
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.1.keytab
--principal=oryx/127.23.242.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:28.255337 32339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:28.255573 32339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:28.255609 32339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:28.256970 32339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:28.257028 32339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:28.257052 32339 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 00:58:28.257072 32339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 00:58:28.258822 32339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.1.keytab
--principal=oryx/127.23.242.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.242.62:39515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:28.259016 32339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:28.259217 32339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:28.265026 32345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:28.265027 32346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:28.265026 32348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:28.265434 32339 server_base.cc:1034] running on GCE node
I20260806 00:58:28.265681 32339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:28.265951 32339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:28.267117 32339 hybrid_clock.cc:648] HybridClock initialized: now 1785977908267079 us; error 59 us; skew 500 ppm
Aug 06 00:58:28 dist-test-slave-dhph krb5kdc[32219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977908, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:28.270448 32339 init.cc:377] Logged in from keytab as oryx/127.23.242.1@KRBTEST.COM (short username oryx)
I20260806 00:58:28.270990 32339 webserver.cc:460] Webserver started at http://127.23.242.1:45563/ using document root <none> and password file <none>
I20260806 00:58:28.271279 32339 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:28.271358 32339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:28.271526 32339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:28.272747 32339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-0/data/instance:
uuid: "51abc5f62ed248a89acb552552d25cf5"
format_stamp: "Formatted at 2026-08-06 00:58:28 on dist-test-slave-dhph"
server_key: "ac8f942d2d20d03eaf23e4fdf70c401d"
server_key_iv: "a2fcc2e5fb83bd85d064266b675ff1c5"
server_key_version: "encryptionkey@0"
I20260806 00:58:28.273492 32339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance:
uuid: "51abc5f62ed248a89acb552552d25cf5"
format_stamp: "Formatted at 2026-08-06 00:58:28 on dist-test-slave-dhph"
server_key: "ac8f942d2d20d03eaf23e4fdf70c401d"
server_key_iv: "a2fcc2e5fb83bd85d064266b675ff1c5"
server_key_version: "encryptionkey@0"
I20260806 00:58:28.275668 32339 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20260806 00:58:28.276680 32355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:28.276954 32339 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:28.277088 32339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "51abc5f62ed248a89acb552552d25cf5"
format_stamp: "Formatted at 2026-08-06 00:58:28 on dist-test-slave-dhph"
server_key: "ac8f942d2d20d03eaf23e4fdf70c401d"
server_key_iv: "a2fcc2e5fb83bd85d064266b675ff1c5"
server_key_version: "encryptionkey@0"
I20260806 00:58:28.277220 32339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:28.289008 32339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:28.291545 32339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:28.291726 32339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:28.292279 32339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:28.292340 32339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:28.292387 32339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:28.292416 32339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:28.300253 32339 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:44129
I20260806 00:58:28.300347 32471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:44129 every 8 connection(s)
Aug 06 00:58:28 dist-test-slave-dhph krb5kdc[32219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977908, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM
I20260806 00:58:28.300683 32339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 00:58:28.303990 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 32339
I20260806 00:58:28.304121 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance
I20260806 00:58:28.304363 24520 external_mini_cluster.cc:1428] Setting key 86a5be07070afa148509ced7dd266a37
I20260806 00:58:28.304623 32364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.293003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:39515 (local address 127.23.242.1:50819)
0806 00:58:28.293189 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:28.293229 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:28.293836 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:28.294167 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:28.294172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:28.294535 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:28.294860 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.294867 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.296178 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.296179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.297696 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.297700 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.298214 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.298215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.298359 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.299187 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:28.299196 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:28.301061 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:28.302999 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:28.303006 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:28.303007 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:28.303235 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:28.303462 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:28.303463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:28.303463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:28.303564 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:28.303828 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:28.303837 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:28.304016 (+ 179us) client_negotiation.cc:769] Sending connection context
0806 00:58:28.304193 (+ 177us) client_negotiation.cc:240] Negotiation successful
0806 00:58:28.304379 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
I20260806 00:58:28.304751 32316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.293014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:50819 (local address 127.23.242.62:39515)
0806 00:58:28.293091 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:28.293093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:28.293316 (+ 223us) server_negotiation.cc:407] Connection header received
0806 00:58:28.293956 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:28.293957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:28.293974 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:28.294035 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:28.294954 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.296056 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.297887 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.298112 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.298185 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.301210 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:28.301212 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:28.301212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:28.301222 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:28.302919 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:28.303345 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:28.303346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:28.303346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:28.303400 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:28.303621 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:28.303622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:28.303622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:28.303736 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:28.303792 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:28.304496 (+ 704us) server_negotiation.cc:299] Negotiation successful
0806 00:58:28.304608 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 00:58:28.305496 32472 heartbeater.cc:347] Connected to a master server at 127.23.242.62:39515
I20260806 00:58:28.305591 32472 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:28.305873 32472 heartbeater.cc:511] Master 127.23.242.62:39515 requested a full tablet report, sending...
I20260806 00:58:28.306480 32266 ts_manager.cc:194] Registered new tserver with Master: 51abc5f62ed248a89acb552552d25cf5 (127.23.242.1:44129)
I20260806 00:58:28.307425 32266 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.242.1@KRBTEST.COM'} at 127.23.242.1:50819
I20260806 00:58:28.313998 32478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.306770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:39515 (local address 127.23.242.1:58971)
0806 00:58:28.306929 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:28.306940 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:28.307030 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:28.307172 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:28.307173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:28.307407 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:28.307490 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.307492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.308692 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.308693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.309328 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.309330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.309766 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.309767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.309843 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.310400 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:28.310405 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:28.310799 (+ 394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:28.312783 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:28.312784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:28.312785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:28.312984 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:28.313238 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:28.313240 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:28.313240 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:28.313310 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:28.313601 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:28.313602 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:28.313666 (+ 64us) client_negotiation.cc:769] Sending connection context
0806 00:58:28.313753 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 00:58:28.313882 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":104,"threads_started":1}
I20260806 00:58:28.313992 32316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.306717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:58971 (local address 127.23.242.62:39515)
0806 00:58:28.306761 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:28.306763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:28.307014 (+ 251us) server_negotiation.cc:407] Connection header received
0806 00:58:28.307096 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:28.307098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:28.307109 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:28.307166 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:28.307698 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.308553 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.309422 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.309664 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.309764 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.310914 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:28.310916 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:28.310916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:28.310924 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:28.312643 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:28.313085 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:28.313086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:28.313087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:28.313154 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:28.313386 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:28.313387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:28.313388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:28.313520 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:28.313611 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:28.313783 (+ 172us) server_negotiation.cc:299] Negotiation successful
0806 00:58:28.313882 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for oryx/127.23.242.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.242.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.242.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.2.keytab.
Entry for principal oryx/127.23.242.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.2.keytab.
I20260806 00:58:28.339737 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:0
--local_ip_for_outbound_sockets=127.23.242.2
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:39515
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.2.keytab
--principal=oryx/127.23.242.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:28.434281 32482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:28.434517 32482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:28.434545 32482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:28.437178 32482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:28.437252 32482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:28.437280 32482 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 00:58:28.437301 32482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 00:58:28.439236 32482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.2.keytab
--principal=oryx/127.23.242.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.242.62:39515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:28.439466 32482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:28.439673 32482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:28.445922 32488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:28.446027 32482 server_base.cc:1034] running on GCE node
W20260806 00:58:28.446141 32490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:28.446183 32487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:28.446506 32482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:28.446846 32482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:28.448091 32482 hybrid_clock.cc:648] HybridClock initialized: now 1785977908448069 us; error 56 us; skew 500 ppm
Aug 06 00:58:28 dist-test-slave-dhph krb5kdc[32219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977908, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:28.451550 32482 init.cc:377] Logged in from keytab as oryx/127.23.242.2@KRBTEST.COM (short username oryx)
I20260806 00:58:28.452492 32482 webserver.cc:460] Webserver started at http://127.23.242.2:45885/ using document root <none> and password file <none>
I20260806 00:58:28.452885 32482 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:28.452972 32482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:28.453161 32482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:28.454797 32482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-1/data/instance:
uuid: "d6065afec92049beb971daffd555d27e"
format_stamp: "Formatted at 2026-08-06 00:58:28 on dist-test-slave-dhph"
server_key: "5ecf5da050db65be4f8efcf63bbed7c8"
server_key_iv: "e41c6048c7b5852486e9aa4bb4661cf9"
server_key_version: "encryptionkey@0"
I20260806 00:58:28.455372 32482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance:
uuid: "d6065afec92049beb971daffd555d27e"
format_stamp: "Formatted at 2026-08-06 00:58:28 on dist-test-slave-dhph"
server_key: "5ecf5da050db65be4f8efcf63bbed7c8"
server_key_iv: "e41c6048c7b5852486e9aa4bb4661cf9"
server_key_version: "encryptionkey@0"
I20260806 00:58:28.457535 32482 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 00:58:28.458966 32497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:28.459467 32482 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:28.459666 32482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "d6065afec92049beb971daffd555d27e"
format_stamp: "Formatted at 2026-08-06 00:58:28 on dist-test-slave-dhph"
server_key: "5ecf5da050db65be4f8efcf63bbed7c8"
server_key_iv: "e41c6048c7b5852486e9aa4bb4661cf9"
server_key_version: "encryptionkey@0"
I20260806 00:58:28.459777 32482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:28.480407 32482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:28.483508 32482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:28.483827 32482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:28.484478 32482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:28.484549 32482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:28.484602 32482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:28.484634 32482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:28.492426 32482 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:41909
Aug 06 00:58:28 dist-test-slave-dhph krb5kdc[32219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977908, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM
I20260806 00:58:28.492506 32615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:41909 every 8 connection(s)
I20260806 00:58:28.493029 32482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 00:58:28.496657 32316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.485275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:48353 (local address 127.23.242.62:39515)
0806 00:58:28.485352 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:28.485355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:28.485615 (+ 260us) server_negotiation.cc:407] Connection header received
0806 00:58:28.486232 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:28.486234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:28.486251 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:28.486346 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:28.487283 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.488146 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.489951 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.490272 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.490361 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.493354 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:28.493357 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:28.493357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:28.493368 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:28.494931 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:28.495425 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:28.495426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:28.495427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:28.495559 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:28.495913 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:28.495914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:28.495915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:28.496066 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:28.496140 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:28.496493 (+ 353us) server_negotiation.cc:299] Negotiation successful
0806 00:58:28.496570 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 00:58:28.496876 32507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.485272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:39515 (local address 127.23.242.2:48353)
0806 00:58:28.485462 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:28.485509 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:28.486100 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:28.486451 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:28.486456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:28.486816 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:28.487164 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.487172 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.488260 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.488261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.489842 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.489846 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.490425 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.490427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.490540 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.491334 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:28.491345 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:28.493114 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:28.495040 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:28.495046 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:28.495048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:28.495315 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:28.495706 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:28.495707 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:28.495707 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:28.495817 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:28.496198 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:28.496205 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:28.496337 (+ 132us) client_negotiation.cc:769] Sending connection context
0806 00:58:28.496430 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 00:58:28.496559 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":89,"threads_started":1}
I20260806 00:58:28.497372 32616 heartbeater.cc:347] Connected to a master server at 127.23.242.62:39515
I20260806 00:58:28.497459 32616 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:28.497712 32616 heartbeater.cc:511] Master 127.23.242.62:39515 requested a full tablet report, sending...
I20260806 00:58:28.498136 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 32482
I20260806 00:58:28.498279 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance
I20260806 00:58:28.498534 24520 external_mini_cluster.cc:1428] Setting key 74e5778a7af14f9465a4d6dc1194fde2
I20260806 00:58:28.499069 32265 ts_manager.cc:194] Registered new tserver with Master: d6065afec92049beb971daffd555d27e (127.23.242.2:41909)
I20260806 00:58:28.499650 32265 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.242.2@KRBTEST.COM'} at 127.23.242.2:48353
I20260806 00:58:28.506199 32622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.498542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:39515 (local address 127.23.242.2:40245)
0806 00:58:28.498778 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:28.498789 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:28.498891 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:28.499059 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:28.499060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:28.499269 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:28.499333 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.499335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.500083 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.500084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.500619 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.500621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.501094 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.501096 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.501196 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.502158 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:28.502176 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:28.502642 (+ 466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:28.504598 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:28.504599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:28.504599 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:28.504823 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:28.505085 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:28.505086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:28.505087 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:28.505153 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:28.505627 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:28.505630 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:28.505926 (+ 296us) client_negotiation.cc:769] Sending connection context
0806 00:58:28.506021 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 00:58:28.506122 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":161,"threads_started":1}
I20260806 00:58:28.506311 32316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.498614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:40245 (local address 127.23.242.62:39515)
0806 00:58:28.498661 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:28.498723 (+ 62us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:28.498872 (+ 149us) server_negotiation.cc:407] Connection header received
0806 00:58:28.498957 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:28.498957 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:28.498990 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:28.499054 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:28.499425 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.499986 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.500737 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.500946 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.501027 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.502730 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:28.502732 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:28.502732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:28.502738 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:28.504492 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:28.504909 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:28.504910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:28.504911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:28.504980 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:28.505232 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:28.505233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:28.505233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:28.505469 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:28.505555 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:28.506076 (+ 521us) server_negotiation.cc:299] Negotiation successful
0806 00:58:28.506173 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
WARNING: no policy specified for oryx/127.23.242.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.242.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.242.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.3.keytab.
Entry for principal oryx/127.23.242.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.3.keytab.
I20260806 00:58:28.539011 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:0
--local_ip_for_outbound_sockets=127.23.242.3
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:39515
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.3.keytab
--principal=oryx/127.23.242.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:28.643472 32625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:28.643841 32625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:28.643910 32625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:28.645463 32625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:28.645557 32625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:28.645689 32625 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 00:58:28.645767 32625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 00:58:28.647855 32625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.3.keytab
--principal=oryx/127.23.242.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.242.62:39515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:28.648212 32625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:28.648499 32625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:28.654882 32631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:28.654990 32633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:28.654990 32630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:28.655417 32625 server_base.cc:1034] running on GCE node
I20260806 00:58:28.655788 32625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:28.656143 32625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:28.657334 32625 hybrid_clock.cc:648] HybridClock initialized: now 1785977908657301 us; error 89 us; skew 500 ppm
Aug 06 00:58:28 dist-test-slave-dhph krb5kdc[32219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977908, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:28.660293 32625 init.cc:377] Logged in from keytab as oryx/127.23.242.3@KRBTEST.COM (short username oryx)
I20260806 00:58:28.660826 32625 webserver.cc:460] Webserver started at http://127.23.242.3:38483/ using document root <none> and password file <none>
I20260806 00:58:28.661044 32625 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:28.661087 32625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:28.661181 32625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:28.662300 32625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-2/data/instance:
uuid: "4ee9f830e89d43f38f5060bf8d613b89"
format_stamp: "Formatted at 2026-08-06 00:58:28 on dist-test-slave-dhph"
server_key: "a1c9425092ff14f3d9d2c4cc4c3f6091"
server_key_iv: "16458bac0c931157d36425b28b32a458"
server_key_version: "encryptionkey@0"
I20260806 00:58:28.662765 32625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance:
uuid: "4ee9f830e89d43f38f5060bf8d613b89"
format_stamp: "Formatted at 2026-08-06 00:58:28 on dist-test-slave-dhph"
server_key: "a1c9425092ff14f3d9d2c4cc4c3f6091"
server_key_iv: "16458bac0c931157d36425b28b32a458"
server_key_version: "encryptionkey@0"
I20260806 00:58:28.664625 32625 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260806 00:58:28.665757 32640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:28.665966 32625 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:28.666071 32625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "4ee9f830e89d43f38f5060bf8d613b89"
format_stamp: "Formatted at 2026-08-06 00:58:28 on dist-test-slave-dhph"
server_key: "a1c9425092ff14f3d9d2c4cc4c3f6091"
server_key_iv: "16458bac0c931157d36425b28b32a458"
server_key_version: "encryptionkey@0"
I20260806 00:58:28.666157 32625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:28.676075 32625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:28.678879 32625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:28.679028 32625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:28.679589 32625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:28.679651 32625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:28.679682 32625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:28.679695 32625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 00:58:28 dist-test-slave-dhph krb5kdc[32219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977908, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM
I20260806 00:58:28.687542 32625 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:36941
I20260806 00:58:28.687656 32757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:36941 every 8 connection(s)
I20260806 00:58:28.688072 32625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
I20260806 00:58:28.691319 32316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.680244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:56153 (local address 127.23.242.62:39515)
0806 00:58:28.680314 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:28.680317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:28.680585 (+ 268us) server_negotiation.cc:407] Connection header received
0806 00:58:28.681111 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:28.681112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:28.681129 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:28.681200 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:28.682240 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.683209 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.684327 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.684709 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.684808 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.688124 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:28.688126 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:28.688126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:28.688135 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:28.689737 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:28.690172 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:28.690173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:28.690173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:28.690239 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:28.690573 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:28.690653 (+ 80us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:28.690654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:28.690800 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:28.690867 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:28.691182 (+ 315us) server_negotiation.cc:299] Negotiation successful
0806 00:58:28.691253 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 00:58:28.691537 32649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.680277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:39515 (local address 127.23.242.3:56153)
0806 00:58:28.680436 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:28.680474 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:28.680993 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:28.681344 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:28.681349 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:28.681765 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:28.682113 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.682129 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.683322 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.683323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.684251 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.684254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.684849 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.684851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.685012 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.685703 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:28.685720 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:28.687942 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:28.689836 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:28.689843 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:28.689845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:28.690088 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:28.690358 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:28.690358 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:28.690359 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:28.690475 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:28.690874 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:28.690880 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:28.691006 (+ 126us) client_negotiation.cc:769] Sending connection context
0806 00:58:28.691106 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 00:58:28.691273 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260806 00:58:28.692386 32758 heartbeater.cc:347] Connected to a master server at 127.23.242.62:39515
I20260806 00:58:28.692499 32758 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:28.692910 32758 heartbeater.cc:511] Master 127.23.242.62:39515 requested a full tablet report, sending...
I20260806 00:58:28.693425 32266 ts_manager.cc:194] Registered new tserver with Master: 4ee9f830e89d43f38f5060bf8d613b89 (127.23.242.3:36941)
I20260806 00:58:28.693917 32266 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.242.3@KRBTEST.COM'} at 127.23.242.3:56153
I20260806 00:58:28.696802 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 32625
I20260806 00:58:28.696924 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance
I20260806 00:58:28.697177 24520 external_mini_cluster.cc:1428] Setting key 8be3687ab8d53ed9f3f8eee666154abb
I20260806 00:58:28.698179 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 00:58:28.701233 32316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.693825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:60991 (local address 127.23.242.62:39515)
0806 00:58:28.693897 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:28.693899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:28.694038 (+ 139us) server_negotiation.cc:407] Connection header received
0806 00:58:28.694104 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:28.694105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:28.694118 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:28.694175 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:28.694534 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.695169 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.696157 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.696369 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.696460 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.697923 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:28.697925 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:28.697925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:28.697932 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:28.699958 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:28.700332 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:28.700334 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:28.700334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:28.700405 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:28.700643 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:28.700644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:28.700645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:28.700811 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:28.700893 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:28.701095 (+ 202us) server_negotiation.cc:299] Negotiation successful
0806 00:58:28.701162 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 00:58:28.701354 32764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.693767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:39515 (local address 127.23.242.3:60991)
0806 00:58:28.693961 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:28.693971 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:28.694051 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:28.694176 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:28.694177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:28.694387 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:28.694449 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.694452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.695373 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.695376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.696017 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.696020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.696561 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.696563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.696678 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.697338 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:28.697343 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:28.697777 (+ 434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:28.700060 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:28.700061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:28.700062 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:28.700252 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:28.700479 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:28.700479 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:28.700479 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:28.700575 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:28.700895 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:28.700897 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:28.700963 (+ 66us) client_negotiation.cc:769] Sending connection context
0806 00:58:28.701053 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 00:58:28.701195 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":94,"threads_started":1}
Aug 06 00:58:28 dist-test-slave-dhph krb5kdc[32219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977908, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 00:58:28 dist-test-slave-dhph krb5kdc[32219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:28 dist-test-slave-dhph krb5kdc[32219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 00:58:28.727985 32316 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:28.720493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34506 (local address 127.23.242.62:39515)
0806 00:58:28.720571 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:28.720574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:28.720590 (+ 16us) server_negotiation.cc:407] Connection header received
0806 00:58:28.720617 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:28.720618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:28.720628 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:28.720687 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:28.721602 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.722194 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.723121 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.723403 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.723482 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.727860 (+ 4378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34506: BlockingRecv error: recv got EOF from 127.0.0.1:34506 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 00:58:28 dist-test-slave-dhph krb5kdc[32219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977908, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM
I20260806 00:58:28.737493 32316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.728957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34520 (local address 127.23.242.62:39515)
0806 00:58:28.729033 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:28.729035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:28.729172 (+ 137us) server_negotiation.cc:407] Connection header received
0806 00:58:28.729235 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:28.729236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:28.729249 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:28.729298 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:28.730000 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.730570 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.731322 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.731519 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.731587 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.734410 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:28.734412 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:28.734412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:28.734419 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:28.736105 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:28.736601 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:28.736602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:28.736603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:28.736670 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:28.736890 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:28.736892 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:28.736892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:28.737027 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:28.737144 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:28.737283 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 00:58:28.737373 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 00:58:28.740345 32266 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 00:58:28.742138 32266 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 00:58:28.750101 313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.745225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:36941 (local address 127.0.0.1:40526)
0806 00:58:28.745592 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:28.745636 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:28.745764 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:28.746104 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:28.746107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:28.746110 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:28.746211 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.746214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.747764 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.747765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.748774 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.748777 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.749723 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.749725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.749831 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.749850 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 00:58:28.749935 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 00:58:28.749992 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":65,"thread_start_us":83,"threads_started":1}
I20260806 00:58:28.750214 314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.745353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40526 (local address 127.23.242.3:36941)
0806 00:58:28.745545 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:28.745550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:28.745712 (+ 162us) server_negotiation.cc:407] Connection header received
0806 00:58:28.745904 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:28.745907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:28.745926 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:28.745987 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:28.746338 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.747646 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.748882 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.749615 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.749708 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.749794 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:28.749968 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 00:58:28.750014 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":122,"threads_started":1}
I20260806 00:58:28.751093 317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.745697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:44129 (local address 127.0.0.1:41310)
0806 00:58:28.746146 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:28.746157 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:28.746205 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:28.746648 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:28.746649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:28.746652 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:28.746694 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.746695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.747657 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.747658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.749171 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.749175 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.750763 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.750764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.750817 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.750820 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 00:58:28.750900 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 00:58:28.750975 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":36,"thread_start_us":93,"threads_started":1}
I20260806 00:58:28.751274 316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.745593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58698 (local address 127.23.242.2:41909)
0806 00:58:28.745925 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:28.745935 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:28.745946 (+ 11us) server_negotiation.cc:407] Connection header received
0806 00:58:28.745991 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:28.745994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:28.746011 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:28.746096 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:28.746321 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.747660 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.749463 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.750482 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.750607 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.750702 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:28.751062 (+ 360us) server_negotiation.cc:299] Negotiation successful
0806 00:58:28.751136 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":180,"threads_started":1}
I20260806 00:58:28.751488 315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.745440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:41909 (local address 127.0.0.1:58698)
0806 00:58:28.745608 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:28.745644 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:28.745791 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:28.746168 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:28.746169 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:28.746171 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:28.746229 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.746231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.747743 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.747744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.749370 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.749372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.750697 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.750700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.750813 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.750820 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 00:58:28.750934 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 00:58:28.750975 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":410,"thread_start_us":86,"threads_started":1}
I20260806 00:58:28.751631 32692 tablet_service.cc:1467] Processing CreateTablet for tablet 511859d0be144d27a3bad02da29deb71 (DEFAULT_TABLE table=test-table [id=80cc91e3897b4fb19ddac48d953fb337]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:28.752033 32692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 511859d0be144d27a3bad02da29deb71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:28.755007 318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.745960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41310 (local address 127.23.242.1:44129)
0806 00:58:28.746404 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:28.746408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:28.746416 (+ 8us) server_negotiation.cc:407] Connection header received
0806 00:58:28.746454 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:28.746455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:28.746469 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:28.746570 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:28.746796 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.747559 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.749279 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.750112 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.750454 (+ 342us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.750724 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:28.750932 (+ 208us) server_negotiation.cc:299] Negotiation successful
0806 00:58:28.750995 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":83,"threads_started":1}
I20260806 00:58:28.755585 32548 tablet_service.cc:1467] Processing CreateTablet for tablet 511859d0be144d27a3bad02da29deb71 (DEFAULT_TABLE table=test-table [id=80cc91e3897b4fb19ddac48d953fb337]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:28.755645 32405 tablet_service.cc:1467] Processing CreateTablet for tablet 511859d0be144d27a3bad02da29deb71 (DEFAULT_TABLE table=test-table [id=80cc91e3897b4fb19ddac48d953fb337]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:28.755995 32548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 511859d0be144d27a3bad02da29deb71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:28.755995 32405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 511859d0be144d27a3bad02da29deb71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:28.756183 319 tablet_bootstrap.cc:492] T 511859d0be144d27a3bad02da29deb71 P 4ee9f830e89d43f38f5060bf8d613b89: Bootstrap starting.
I20260806 00:58:28.757161 319 tablet_bootstrap.cc:654] T 511859d0be144d27a3bad02da29deb71 P 4ee9f830e89d43f38f5060bf8d613b89: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:28.757572 319 log.cc:824] T 511859d0be144d27a3bad02da29deb71 P 4ee9f830e89d43f38f5060bf8d613b89: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:28.758453 319 tablet_bootstrap.cc:492] T 511859d0be144d27a3bad02da29deb71 P 4ee9f830e89d43f38f5060bf8d613b89: No bootstrap required, opened a new log
I20260806 00:58:28.758551 319 ts_tablet_manager.cc:1397] T 511859d0be144d27a3bad02da29deb71 P 4ee9f830e89d43f38f5060bf8d613b89: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 00:58:28.761034 321 tablet_bootstrap.cc:492] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e: Bootstrap starting.
I20260806 00:58:28.761415 319 raft_consensus.cc:348] T 511859d0be144d27a3bad02da29deb71 P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } }
I20260806 00:58:28.761624 319 raft_consensus.cc:374] T 511859d0be144d27a3bad02da29deb71 P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:28.761668 319 raft_consensus.cc:729] T 511859d0be144d27a3bad02da29deb71 P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee9f830e89d43f38f5060bf8d613b89, State: Initialized, Role: FOLLOWER
I20260806 00:58:28.761824 319 consensus_queue.cc:260] T 511859d0be144d27a3bad02da29deb71 P 4ee9f830e89d43f38f5060bf8d613b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } }
I20260806 00:58:28.762146 319 ts_tablet_manager.cc:1428] T 511859d0be144d27a3bad02da29deb71 P 4ee9f830e89d43f38f5060bf8d613b89: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260806 00:58:28.762187 321 tablet_bootstrap.cc:654] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:28.762185 32758 heartbeater.cc:503] Master 127.23.242.62:39515 was elected leader, sending a full tablet report...
I20260806 00:58:28.762501 321 log.cc:824] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:28.763079 324 tablet_bootstrap.cc:492] T 511859d0be144d27a3bad02da29deb71 P 51abc5f62ed248a89acb552552d25cf5: Bootstrap starting.
I20260806 00:58:28.763854 321 tablet_bootstrap.cc:492] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e: No bootstrap required, opened a new log
I20260806 00:58:28.763994 321 ts_tablet_manager.cc:1397] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 00:58:28.764952 324 tablet_bootstrap.cc:654] T 511859d0be144d27a3bad02da29deb71 P 51abc5f62ed248a89acb552552d25cf5: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:28.765296 324 log.cc:824] T 511859d0be144d27a3bad02da29deb71 P 51abc5f62ed248a89acb552552d25cf5: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:28.766204 324 tablet_bootstrap.cc:492] T 511859d0be144d27a3bad02da29deb71 P 51abc5f62ed248a89acb552552d25cf5: No bootstrap required, opened a new log
I20260806 00:58:28.766283 324 ts_tablet_manager.cc:1397] T 511859d0be144d27a3bad02da29deb71 P 51abc5f62ed248a89acb552552d25cf5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 00:58:28.767256 321 raft_consensus.cc:348] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } }
I20260806 00:58:28.767416 321 raft_consensus.cc:374] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:28.767463 321 raft_consensus.cc:729] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6065afec92049beb971daffd555d27e, State: Initialized, Role: FOLLOWER
I20260806 00:58:28.767593 321 consensus_queue.cc:260] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } }
I20260806 00:58:28.767932 321 ts_tablet_manager.cc:1428] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260806 00:58:28.767992 32616 heartbeater.cc:503] Master 127.23.242.62:39515 was elected leader, sending a full tablet report...
I20260806 00:58:28.768658 324 raft_consensus.cc:348] T 511859d0be144d27a3bad02da29deb71 P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } }
I20260806 00:58:28.768867 324 raft_consensus.cc:374] T 511859d0be144d27a3bad02da29deb71 P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:28.768918 324 raft_consensus.cc:729] T 511859d0be144d27a3bad02da29deb71 P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51abc5f62ed248a89acb552552d25cf5, State: Initialized, Role: FOLLOWER
I20260806 00:58:28.769068 324 consensus_queue.cc:260] T 511859d0be144d27a3bad02da29deb71 P 51abc5f62ed248a89acb552552d25cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } }
I20260806 00:58:28.769407 324 ts_tablet_manager.cc:1428] T 511859d0be144d27a3bad02da29deb71 P 51abc5f62ed248a89acb552552d25cf5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 00:58:28.769419 32472 heartbeater.cc:503] Master 127.23.242.62:39515 was elected leader, sending a full tablet report...
I20260806 00:58:28.774791 327 raft_consensus.cc:482] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 00:58:28.774912 327 raft_consensus.cc:504] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } }
I20260806 00:58:28.775265 327 leader_election.cc:288] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ee9f830e89d43f38f5060bf8d613b89 (127.23.242.3:36941), 51abc5f62ed248a89acb552552d25cf5 (127.23.242.1:44129)
I20260806 00:58:28.778918 314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.775519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:54083 (local address 127.23.242.3:36941)
0806 00:58:28.775562 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:28.775564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:28.775578 (+ 14us) server_negotiation.cc:407] Connection header received
0806 00:58:28.775753 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:28.775754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:28.775765 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:28.775841 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:28.776014 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.776747 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.777743 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.778409 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.778503 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.778514 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:28.778821 (+ 307us) server_negotiation.cc:299] Negotiation successful
0806 00:58:28.778859 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 00:58:28.778934 32507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.775473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:36941 (local address 127.23.242.2:54083)
0806 00:58:28.775534 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:28.775548 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:28.775611 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:28.775861 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:28.775862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:28.775864 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:28.775940 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.775942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.776829 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.776831 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.777634 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.777636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.778522 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.778523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.778679 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.778684 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 00:58:28.778803 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 00:58:28.778854 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260806 00:58:28.779275 32712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "511859d0be144d27a3bad02da29deb71" candidate_uuid: "d6065afec92049beb971daffd555d27e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee9f830e89d43f38f5060bf8d613b89" is_pre_election: true
I20260806 00:58:28.779464 32712 raft_consensus.cc:2454] T 511859d0be144d27a3bad02da29deb71 P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6065afec92049beb971daffd555d27e in term 0.
I20260806 00:58:28.779706 318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.775692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:44213 (local address 127.23.242.1:44129)
0806 00:58:28.775743 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:28.775745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:28.775902 (+ 157us) server_negotiation.cc:407] Connection header received
0806 00:58:28.775974 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:28.775976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:28.775987 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:28.776048 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:28.776275 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.776855 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.778728 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.779273 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.779353 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.779362 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:28.779586 (+ 224us) server_negotiation.cc:299] Negotiation successful
0806 00:58:28.779635 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 00:58:28.779758 32499 leader_election.cc:302] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ee9f830e89d43f38f5060bf8d613b89, d6065afec92049beb971daffd555d27e; no voters:
I20260806 00:58:28.779860 329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.775617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:44129 (local address 127.23.242.2:44213)
0806 00:58:28.775843 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:28.775854 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:28.775911 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:28.776052 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:28.776052 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:28.776053 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:28.776114 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.776115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.777076 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.777077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.778544 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.778547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.779369 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.779371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.779456 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.779461 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 00:58:28.779530 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 00:58:28.779684 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":102,"threads_started":1}
I20260806 00:58:28.779995 327 raft_consensus.cc:2781] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 00:58:28.780068 327 raft_consensus.cc:482] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 00:58:28.780098 327 raft_consensus.cc:3037] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:28.780253 32425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "511859d0be144d27a3bad02da29deb71" candidate_uuid: "d6065afec92049beb971daffd555d27e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51abc5f62ed248a89acb552552d25cf5" is_pre_election: true
I20260806 00:58:28.780411 32425 raft_consensus.cc:2454] T 511859d0be144d27a3bad02da29deb71 P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6065afec92049beb971daffd555d27e in term 0.
I20260806 00:58:28.781327 327 raft_consensus.cc:504] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } }
I20260806 00:58:28.781478 327 leader_election.cc:288] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e [CANDIDATE]: Term 1 election: Requested vote from peers 4ee9f830e89d43f38f5060bf8d613b89 (127.23.242.3:36941), 51abc5f62ed248a89acb552552d25cf5 (127.23.242.1:44129)
I20260806 00:58:28.781710 32425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "511859d0be144d27a3bad02da29deb71" candidate_uuid: "d6065afec92049beb971daffd555d27e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51abc5f62ed248a89acb552552d25cf5"
I20260806 00:58:28.781806 32425 raft_consensus.cc:3037] T 511859d0be144d27a3bad02da29deb71 P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:28.782670 32425 raft_consensus.cc:2454] T 511859d0be144d27a3bad02da29deb71 P 51abc5f62ed248a89acb552552d25cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6065afec92049beb971daffd555d27e in term 1.
I20260806 00:58:28.782738 32712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "511859d0be144d27a3bad02da29deb71" candidate_uuid: "d6065afec92049beb971daffd555d27e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee9f830e89d43f38f5060bf8d613b89"
I20260806 00:58:28.782843 32499 leader_election.cc:302] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51abc5f62ed248a89acb552552d25cf5, d6065afec92049beb971daffd555d27e; no voters:
I20260806 00:58:28.782841 32712 raft_consensus.cc:3037] T 511859d0be144d27a3bad02da29deb71 P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:28.783070 327 raft_consensus.cc:2781] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:28.783272 327 raft_consensus.cc:686] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e [term 1 LEADER]: Becoming Leader. State: Replica: d6065afec92049beb971daffd555d27e, State: Running, Role: LEADER
I20260806 00:58:28.783468 327 consensus_queue.cc:237] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } }
I20260806 00:58:28.783859 32712 raft_consensus.cc:2454] T 511859d0be144d27a3bad02da29deb71 P 4ee9f830e89d43f38f5060bf8d613b89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6065afec92049beb971daffd555d27e in term 1.
I20260806 00:58:28.784759 32266 catalog_manager.cc:5495] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e reported cstate change: term changed from 0 to 1, leader changed from <none> to d6065afec92049beb971daffd555d27e (127.23.242.2). New cstate: current_term: 1 leader_uuid: "d6065afec92049beb971daffd555d27e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } health_report { overall_health: UNKNOWN } } }
I20260806 00:58:28.797075 316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.794117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58700 (local address 127.23.242.2:41909)
0806 00:58:28.794191 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:28.794193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:28.794203 (+ 10us) server_negotiation.cc:407] Connection header received
0806 00:58:28.794272 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:28.794272 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:28.794290 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:28.794363 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 00:58:28.794499 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.795172 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.796006 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.796190 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.796251 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.796490 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 00:58:28.796671 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 00:58:28.796747 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:28.796875 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 00:58:28.796933 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 00:58:28.799427 32425 raft_consensus.cc:1261] T 511859d0be144d27a3bad02da29deb71 P 51abc5f62ed248a89acb552552d25cf5 [term 1 FOLLOWER]: Refusing update from remote peer d6065afec92049beb971daffd555d27e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:28.799424 32712 raft_consensus.cc:1261] T 511859d0be144d27a3bad02da29deb71 P 4ee9f830e89d43f38f5060bf8d613b89 [term 1 FOLLOWER]: Refusing update from remote peer d6065afec92049beb971daffd555d27e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:28.799880 330 consensus_queue.cc:1035] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:28.800030 327 consensus_queue.cc:1035] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 00:58:28.801445 32473 tablet.cc:2369] T 511859d0be144d27a3bad02da29deb71 P 51abc5f62ed248a89acb552552d25cf5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 00:58:28.803263 334 mvcc.cc:205] Tried to move back new op lower bound from 7315365514438709248 to 7315365514378850304. Current Snapshot: MvccSnapshot[applied={T|T < 7315365514438709248 or (T in {7315365514438709248})}]
I20260806 00:58:28.803388 335 mvcc.cc:205] Tried to move back new op lower bound from 7315365514438709248 to 7315365514378850304. Current Snapshot: MvccSnapshot[applied={T|T < 7315365514438709248 or (T in {7315365514438709248})}]
I20260806 00:58:28.804982 32266 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:58:28.805295 32266 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:58:28.806504 32266 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6398079945c47f6bc6b7163ae969179: Sending DeleteTablet for 3 replicas of tablet 511859d0be144d27a3bad02da29deb71
I20260806 00:58:28.806895 32692 tablet_service.cc:1514] Processing DeleteTablet for tablet 511859d0be144d27a3bad02da29deb71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:28 UTC) from {username='oryx', principal='oryx/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:40526
I20260806 00:58:28.806942 32548 tablet_service.cc:1514] Processing DeleteTablet for tablet 511859d0be144d27a3bad02da29deb71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:28 UTC) from {username='oryx', principal='oryx/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:58698
I20260806 00:58:28.806983 32405 tablet_service.cc:1514] Processing DeleteTablet for tablet 511859d0be144d27a3bad02da29deb71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:28 UTC) from {username='oryx', principal='oryx/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:41310
I20260806 00:58:28.807168 345 tablet_replica.cc:330] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e: stopping tablet replica
I20260806 00:58:28.807237 343 tablet_replica.cc:330] T 511859d0be144d27a3bad02da29deb71 P 4ee9f830e89d43f38f5060bf8d613b89: stopping tablet replica
I20260806 00:58:28.807260 345 raft_consensus.cc:2229] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e [term 1 LEADER]: Raft consensus shutting down.
I20260806 00:58:28.807349 345 raft_consensus.cc:2258] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:28.807449 32266 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 00:58:28.807618 32266 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 00:58:28.807866 345 ts_tablet_manager.cc:1905] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:28.808110 343 raft_consensus.cc:2229] T 511859d0be144d27a3bad02da29deb71 P 4ee9f830e89d43f38f5060bf8d613b89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:28.809108 32548 tablet_service.cc:1467] Processing CreateTablet for tablet 1096d16cd8e34d8bac856204dd068144 (DEFAULT_TABLE table=test-table [id=22aac58c04b94741b0ccb43248c37fd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:28.809098 32692 tablet_service.cc:1467] Processing CreateTablet for tablet 1096d16cd8e34d8bac856204dd068144 (DEFAULT_TABLE table=test-table [id=22aac58c04b94741b0ccb43248c37fd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:28.809228 32548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1096d16cd8e34d8bac856204dd068144. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:28.809228 32692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1096d16cd8e34d8bac856204dd068144. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:28.809942 345 ts_tablet_manager.cc:1918] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 00:58:28.810011 345 log.cc:1196] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-1/wal/wals/511859d0be144d27a3bad02da29deb71
I20260806 00:58:28.810303 345 ts_tablet_manager.cc:1939] T 511859d0be144d27a3bad02da29deb71 P d6065afec92049beb971daffd555d27e: Deleting consensus metadata
I20260806 00:58:28.810736 32250 catalog_manager.cc:4841] TS d6065afec92049beb971daffd555d27e (127.23.242.2:41909): tablet 511859d0be144d27a3bad02da29deb71 (table test-table [id=80cc91e3897b4fb19ddac48d953fb337]) successfully deleted
I20260806 00:58:28.810768 32405 tablet_service.cc:1467] Processing CreateTablet for tablet 1096d16cd8e34d8bac856204dd068144 (DEFAULT_TABLE table=test-table [id=22aac58c04b94741b0ccb43248c37fd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:28.810817 319 tablet_bootstrap.cc:492] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89: Bootstrap starting.
I20260806 00:58:28.810897 32405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1096d16cd8e34d8bac856204dd068144. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:28.811285 319 tablet_bootstrap.cc:654] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:28.812009 319 tablet_bootstrap.cc:492] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89: No bootstrap required, opened a new log
I20260806 00:58:28.812071 319 ts_tablet_manager.cc:1397] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:28.812476 324 tablet_bootstrap.cc:492] T 1096d16cd8e34d8bac856204dd068144 P 51abc5f62ed248a89acb552552d25cf5: Bootstrap starting.
I20260806 00:58:28.812973 324 tablet_bootstrap.cc:654] T 1096d16cd8e34d8bac856204dd068144 P 51abc5f62ed248a89acb552552d25cf5: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:28.813035 321 tablet_bootstrap.cc:492] T 1096d16cd8e34d8bac856204dd068144 P d6065afec92049beb971daffd555d27e: Bootstrap starting.
I20260806 00:58:28.813539 321 tablet_bootstrap.cc:654] T 1096d16cd8e34d8bac856204dd068144 P d6065afec92049beb971daffd555d27e: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:28.814225 324 tablet_bootstrap.cc:492] T 1096d16cd8e34d8bac856204dd068144 P 51abc5f62ed248a89acb552552d25cf5: No bootstrap required, opened a new log
I20260806 00:58:28.814293 324 ts_tablet_manager.cc:1397] T 1096d16cd8e34d8bac856204dd068144 P 51abc5f62ed248a89acb552552d25cf5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 00:58:28.814244 319 raft_consensus.cc:348] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } }
I20260806 00:58:28.814334 319 raft_consensus.cc:374] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:28.814373 319 raft_consensus.cc:729] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee9f830e89d43f38f5060bf8d613b89, State: Initialized, Role: FOLLOWER
I20260806 00:58:28.814424 319 consensus_queue.cc:260] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } }
I20260806 00:58:28.814429 324 raft_consensus.cc:348] T 1096d16cd8e34d8bac856204dd068144 P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } }
I20260806 00:58:28.814498 324 raft_consensus.cc:374] T 1096d16cd8e34d8bac856204dd068144 P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:28.814523 324 raft_consensus.cc:729] T 1096d16cd8e34d8bac856204dd068144 P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51abc5f62ed248a89acb552552d25cf5, State: Initialized, Role: FOLLOWER
I20260806 00:58:28.814556 319 ts_tablet_manager.cc:1428] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260806 00:58:28.814612 324 consensus_queue.cc:260] T 1096d16cd8e34d8bac856204dd068144 P 51abc5f62ed248a89acb552552d25cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } }
I20260806 00:58:28.814749 324 ts_tablet_manager.cc:1428] T 1096d16cd8e34d8bac856204dd068144 P 51abc5f62ed248a89acb552552d25cf5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260806 00:58:28.815442 321 tablet_bootstrap.cc:492] T 1096d16cd8e34d8bac856204dd068144 P d6065afec92049beb971daffd555d27e: No bootstrap required, opened a new log
I20260806 00:58:28.815503 321 ts_tablet_manager.cc:1397] T 1096d16cd8e34d8bac856204dd068144 P d6065afec92049beb971daffd555d27e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 00:58:28.815629 321 raft_consensus.cc:348] T 1096d16cd8e34d8bac856204dd068144 P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } }
I20260806 00:58:28.815688 321 raft_consensus.cc:374] T 1096d16cd8e34d8bac856204dd068144 P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:28.815703 321 raft_consensus.cc:729] T 1096d16cd8e34d8bac856204dd068144 P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6065afec92049beb971daffd555d27e, State: Initialized, Role: FOLLOWER
I20260806 00:58:28.815770 321 consensus_queue.cc:260] T 1096d16cd8e34d8bac856204dd068144 P d6065afec92049beb971daffd555d27e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } }
I20260806 00:58:28.815878 321 ts_tablet_manager.cc:1428] T 1096d16cd8e34d8bac856204dd068144 P d6065afec92049beb971daffd555d27e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:28.816089 343 raft_consensus.cc:2258] T 511859d0be144d27a3bad02da29deb71 P 4ee9f830e89d43f38f5060bf8d613b89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:28.816613 343 ts_tablet_manager.cc:1905] T 511859d0be144d27a3bad02da29deb71 P 4ee9f830e89d43f38f5060bf8d613b89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:28.818681 343 ts_tablet_manager.cc:1918] T 511859d0be144d27a3bad02da29deb71 P 4ee9f830e89d43f38f5060bf8d613b89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 00:58:28.818773 343 log.cc:1196] T 511859d0be144d27a3bad02da29deb71 P 4ee9f830e89d43f38f5060bf8d613b89: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-2/wal/wals/511859d0be144d27a3bad02da29deb71
I20260806 00:58:28.819134 343 ts_tablet_manager.cc:1939] T 511859d0be144d27a3bad02da29deb71 P 4ee9f830e89d43f38f5060bf8d613b89: Deleting consensus metadata
I20260806 00:58:28.819590 32250 catalog_manager.cc:4841] TS 4ee9f830e89d43f38f5060bf8d613b89 (127.23.242.3:36941): tablet 511859d0be144d27a3bad02da29deb71 (table test-table [id=80cc91e3897b4fb19ddac48d953fb337]) successfully deleted
I20260806 00:58:28.822715 344 tablet_replica.cc:330] T 511859d0be144d27a3bad02da29deb71 P 51abc5f62ed248a89acb552552d25cf5: stopping tablet replica
I20260806 00:58:28.822849 344 raft_consensus.cc:2229] T 511859d0be144d27a3bad02da29deb71 P 51abc5f62ed248a89acb552552d25cf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:28.822944 344 raft_consensus.cc:2258] T 511859d0be144d27a3bad02da29deb71 P 51abc5f62ed248a89acb552552d25cf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:28.823433 344 ts_tablet_manager.cc:1905] T 511859d0be144d27a3bad02da29deb71 P 51abc5f62ed248a89acb552552d25cf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:28.825645 344 ts_tablet_manager.cc:1918] T 511859d0be144d27a3bad02da29deb71 P 51abc5f62ed248a89acb552552d25cf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 00:58:28.825742 344 log.cc:1196] T 511859d0be144d27a3bad02da29deb71 P 51abc5f62ed248a89acb552552d25cf5: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-0/wal/wals/511859d0be144d27a3bad02da29deb71
I20260806 00:58:28.826016 344 ts_tablet_manager.cc:1939] T 511859d0be144d27a3bad02da29deb71 P 51abc5f62ed248a89acb552552d25cf5: Deleting consensus metadata
I20260806 00:58:28.826738 32250 catalog_manager.cc:4841] TS 51abc5f62ed248a89acb552552d25cf5 (127.23.242.1:44129): tablet 511859d0be144d27a3bad02da29deb71 (table test-table [id=80cc91e3897b4fb19ddac48d953fb337]) successfully deleted
I20260806 00:58:28.847129 322 raft_consensus.cc:482] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 00:58:28.847273 322 raft_consensus.cc:504] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } }
I20260806 00:58:28.847630 322 leader_election.cc:288] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51abc5f62ed248a89acb552552d25cf5 (127.23.242.1:44129), d6065afec92049beb971daffd555d27e (127.23.242.2:41909)
I20260806 00:58:28.851908 32649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.847844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:44129 (local address 127.23.242.3:45523)
0806 00:58:28.847915 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:28.847931 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:28.848007 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:28.848340 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:28.848341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:28.848343 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:28.848449 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.848452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.849639 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.849640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.850955 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.850957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.851576 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.851577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.851664 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.851668 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 00:58:28.851731 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 00:58:28.851782 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":32}
I20260806 00:58:28.851917 318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.847944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:45523 (local address 127.23.242.1:44129)
0806 00:58:28.847994 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:28.847996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:28.848008 (+ 12us) server_negotiation.cc:407] Connection header received
0806 00:58:28.848199 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:28.848201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:28.848215 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:28.848280 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:28.848543 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.849535 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.851082 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.851475 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.851638 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.851651 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:28.851792 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 00:58:28.851841 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260806 00:58:28.852030 316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.848097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:59345 (local address 127.23.242.2:41909)
0806 00:58:28.848154 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:28.848156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:28.848237 (+ 81us) server_negotiation.cc:407] Connection header received
0806 00:58:28.848306 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:28.848306 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:28.848316 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:28.848381 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:28.848642 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.849354 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.851043 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.851645 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.851730 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.851739 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:28.851898 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 00:58:28.851950 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 00:58:28.852289 349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.847948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:41909 (local address 127.23.242.3:59345)
0806 00:58:28.848074 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:28.848085 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:28.848240 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:28.848391 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:28.848392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:28.848393 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:28.848466 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.848467 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.849454 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.849455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.850942 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.850947 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.851729 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.851730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.851798 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.851802 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 00:58:28.852124 (+ 322us) client_negotiation.cc:240] Negotiation successful
0806 00:58:28.852188 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":2,"thread_start_us":57,"threads_started":1}
I20260806 00:58:28.852671 32425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1096d16cd8e34d8bac856204dd068144" candidate_uuid: "4ee9f830e89d43f38f5060bf8d613b89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51abc5f62ed248a89acb552552d25cf5" is_pre_election: true
I20260806 00:58:28.852695 32568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1096d16cd8e34d8bac856204dd068144" candidate_uuid: "4ee9f830e89d43f38f5060bf8d613b89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6065afec92049beb971daffd555d27e" is_pre_election: true
I20260806 00:58:28.852797 32425 raft_consensus.cc:2454] T 1096d16cd8e34d8bac856204dd068144 P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ee9f830e89d43f38f5060bf8d613b89 in term 0.
I20260806 00:58:28.852862 32568 raft_consensus.cc:2454] T 1096d16cd8e34d8bac856204dd068144 P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ee9f830e89d43f38f5060bf8d613b89 in term 0.
I20260806 00:58:28.853014 32642 leader_election.cc:302] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ee9f830e89d43f38f5060bf8d613b89, 51abc5f62ed248a89acb552552d25cf5; no voters:
I20260806 00:58:28.853204 322 raft_consensus.cc:2781] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 00:58:28.853260 322 raft_consensus.cc:482] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 00:58:28.853293 322 raft_consensus.cc:3037] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:28.854188 322 raft_consensus.cc:504] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } }
I20260806 00:58:28.854418 322 leader_election.cc:288] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89 [CANDIDATE]: Term 1 election: Requested vote from peers 51abc5f62ed248a89acb552552d25cf5 (127.23.242.1:44129), d6065afec92049beb971daffd555d27e (127.23.242.2:41909)
I20260806 00:58:28.854557 32425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1096d16cd8e34d8bac856204dd068144" candidate_uuid: "4ee9f830e89d43f38f5060bf8d613b89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51abc5f62ed248a89acb552552d25cf5"
I20260806 00:58:28.854632 32568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1096d16cd8e34d8bac856204dd068144" candidate_uuid: "4ee9f830e89d43f38f5060bf8d613b89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6065afec92049beb971daffd555d27e"
I20260806 00:58:28.854676 32425 raft_consensus.cc:3037] T 1096d16cd8e34d8bac856204dd068144 P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:28.854702 32568 raft_consensus.cc:3037] T 1096d16cd8e34d8bac856204dd068144 P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:28.855357 32425 raft_consensus.cc:2454] T 1096d16cd8e34d8bac856204dd068144 P 51abc5f62ed248a89acb552552d25cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ee9f830e89d43f38f5060bf8d613b89 in term 1.
I20260806 00:58:28.855549 32568 raft_consensus.cc:2454] T 1096d16cd8e34d8bac856204dd068144 P d6065afec92049beb971daffd555d27e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ee9f830e89d43f38f5060bf8d613b89 in term 1.
I20260806 00:58:28.855582 32642 leader_election.cc:302] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ee9f830e89d43f38f5060bf8d613b89, 51abc5f62ed248a89acb552552d25cf5; no voters:
I20260806 00:58:28.855734 322 raft_consensus.cc:2781] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:28.855870 322 raft_consensus.cc:686] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89 [term 1 LEADER]: Becoming Leader. State: Replica: 4ee9f830e89d43f38f5060bf8d613b89, State: Running, Role: LEADER
I20260806 00:58:28.855952 322 consensus_queue.cc:237] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } }
I20260806 00:58:28.856758 32265 catalog_manager.cc:5495] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ee9f830e89d43f38f5060bf8d613b89 (127.23.242.3). New cstate: current_term: 1 leader_uuid: "4ee9f830e89d43f38f5060bf8d613b89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } health_report { overall_health: HEALTHY } } }
Aug 06 00:58:28 dist-test-slave-dhph krb5kdc[32219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977908, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.62@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM
I20260806 00:58:28.896179 357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.887482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:39515 (local address 127.0.0.1:34526)
0806 00:58:28.887661 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:28.887680 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:28.887746 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:28.887901 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:28.887902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:28.888240 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:28.888300 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.888303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.889305 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.889306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.890122 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.890133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.890709 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.890711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.890782 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.891426 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:28.891431 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:28.892846 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:28.894883 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:28.894891 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:28.894894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:28.895131 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:28.895383 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:28.895384 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:28.895384 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:28.895457 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:28.895820 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:28.895827 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:28.895879 (+ 52us) client_negotiation.cc:769] Sending connection context
0806 00:58:28.895957 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 00:58:28.896089 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":111,"threads_started":1}
I20260806 00:58:28.897060 32316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.887572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34526 (local address 127.23.242.62:39515)
0806 00:58:28.887635 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:28.887637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:28.887755 (+ 118us) server_negotiation.cc:407] Connection header received
0806 00:58:28.887795 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:28.887796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:28.887808 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:28.887898 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:28.888399 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.889199 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.890290 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.890615 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.890700 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.892949 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:28.892950 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:28.892950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:28.892958 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:28.894764 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:28.895225 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:28.895226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:28.895227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:28.895304 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:28.895535 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:28.895536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:28.895536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:28.895688 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:28.895768 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:28.896013 (+ 245us) server_negotiation.cc:299] Negotiation successful
0806 00:58:28.896175 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":35}
I20260806 00:58:28.899817 32265 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:34526:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 00:58:28.900090 32265 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 00:58:28.901710 32548 tablet_service.cc:1467] Processing CreateTablet for tablet af590ebaadfa4d6c8d65037db211fe6a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4b619e98f0744888b4b53c018e98e7e9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 00:58:28.901741 32692 tablet_service.cc:1467] Processing CreateTablet for tablet af590ebaadfa4d6c8d65037db211fe6a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4b619e98f0744888b4b53c018e98e7e9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 00:58:28.901762 32405 tablet_service.cc:1467] Processing CreateTablet for tablet af590ebaadfa4d6c8d65037db211fe6a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4b619e98f0744888b4b53c018e98e7e9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 00:58:28.901885 32548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af590ebaadfa4d6c8d65037db211fe6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:28.901883 32692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af590ebaadfa4d6c8d65037db211fe6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:28.901921 32405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af590ebaadfa4d6c8d65037db211fe6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:28.903502 319 tablet_bootstrap.cc:492] T af590ebaadfa4d6c8d65037db211fe6a P 4ee9f830e89d43f38f5060bf8d613b89: Bootstrap starting.
I20260806 00:58:28.903497 321 tablet_bootstrap.cc:492] T af590ebaadfa4d6c8d65037db211fe6a P d6065afec92049beb971daffd555d27e: Bootstrap starting.
I20260806 00:58:28.903621 324 tablet_bootstrap.cc:492] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5: Bootstrap starting.
I20260806 00:58:28.903970 319 tablet_bootstrap.cc:654] T af590ebaadfa4d6c8d65037db211fe6a P 4ee9f830e89d43f38f5060bf8d613b89: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:28.904052 324 tablet_bootstrap.cc:654] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:28.904196 321 tablet_bootstrap.cc:654] T af590ebaadfa4d6c8d65037db211fe6a P d6065afec92049beb971daffd555d27e: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:28.904778 319 tablet_bootstrap.cc:492] T af590ebaadfa4d6c8d65037db211fe6a P 4ee9f830e89d43f38f5060bf8d613b89: No bootstrap required, opened a new log
I20260806 00:58:28.904829 319 ts_tablet_manager.cc:1397] T af590ebaadfa4d6c8d65037db211fe6a P 4ee9f830e89d43f38f5060bf8d613b89: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:28.904910 324 tablet_bootstrap.cc:492] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5: No bootstrap required, opened a new log
I20260806 00:58:28.904966 324 ts_tablet_manager.cc:1397] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:28.904990 319 raft_consensus.cc:348] T af590ebaadfa4d6c8d65037db211fe6a P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } }
I20260806 00:58:28.905081 319 raft_consensus.cc:374] T af590ebaadfa4d6c8d65037db211fe6a P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:28.905114 319 raft_consensus.cc:729] T af590ebaadfa4d6c8d65037db211fe6a P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee9f830e89d43f38f5060bf8d613b89, State: Initialized, Role: FOLLOWER
I20260806 00:58:28.905102 324 raft_consensus.cc:348] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } }
I20260806 00:58:28.905172 324 raft_consensus.cc:374] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:28.905153 319 consensus_queue.cc:260] T af590ebaadfa4d6c8d65037db211fe6a P 4ee9f830e89d43f38f5060bf8d613b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } }
I20260806 00:58:28.905202 324 raft_consensus.cc:729] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51abc5f62ed248a89acb552552d25cf5, State: Initialized, Role: FOLLOWER
I20260806 00:58:28.905251 319 ts_tablet_manager.cc:1428] T af590ebaadfa4d6c8d65037db211fe6a P 4ee9f830e89d43f38f5060bf8d613b89: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:28.905256 324 consensus_queue.cc:260] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } }
I20260806 00:58:28.905345 324 ts_tablet_manager.cc:1428] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:28.905537 322 tablet_replica.cc:439] T af590ebaadfa4d6c8d65037db211fe6a P 4ee9f830e89d43f38f5060bf8d613b89: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } }
I20260806 00:58:28.905640 322 tablet_replica.cc:442] T af590ebaadfa4d6c8d65037db211fe6a P 4ee9f830e89d43f38f5060bf8d613b89: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 00:58:28.905814 321 tablet_bootstrap.cc:492] T af590ebaadfa4d6c8d65037db211fe6a P d6065afec92049beb971daffd555d27e: No bootstrap required, opened a new log
I20260806 00:58:28.905862 321 ts_tablet_manager.cc:1397] T af590ebaadfa4d6c8d65037db211fe6a P d6065afec92049beb971daffd555d27e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 00:58:28.905897 328 tablet_replica.cc:439] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } }
I20260806 00:58:28.905992 328 tablet_replica.cc:442] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 00:58:28.906013 321 raft_consensus.cc:348] T af590ebaadfa4d6c8d65037db211fe6a P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } }
I20260806 00:58:28.906127 321 raft_consensus.cc:374] T af590ebaadfa4d6c8d65037db211fe6a P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:28.906162 321 raft_consensus.cc:729] T af590ebaadfa4d6c8d65037db211fe6a P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6065afec92049beb971daffd555d27e, State: Initialized, Role: FOLLOWER
I20260806 00:58:28.906212 321 consensus_queue.cc:260] T af590ebaadfa4d6c8d65037db211fe6a P d6065afec92049beb971daffd555d27e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } }
I20260806 00:58:28.906335 321 ts_tablet_manager.cc:1428] T af590ebaadfa4d6c8d65037db211fe6a P d6065afec92049beb971daffd555d27e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:28.906430 338 tablet_replica.cc:439] T af590ebaadfa4d6c8d65037db211fe6a P d6065afec92049beb971daffd555d27e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } }
I20260806 00:58:28.906497 338 tablet_replica.cc:442] T af590ebaadfa4d6c8d65037db211fe6a P d6065afec92049beb971daffd555d27e: This TxnStatusTablet replica's current role is: FOLLOWER
W20260806 00:58:28.938735 32759 tablet.cc:2369] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 00:58:28.940748 328 raft_consensus.cc:482] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 00:58:28.940845 328 raft_consensus.cc:504] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } }
I20260806 00:58:28.941238 328 leader_election.cc:288] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6065afec92049beb971daffd555d27e (127.23.242.2:41909), 4ee9f830e89d43f38f5060bf8d613b89 (127.23.242.3:36941)
I20260806 00:58:28.945253 316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.941587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:46367 (local address 127.23.242.2:41909)
0806 00:58:28.941688 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:28.941690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:28.941762 (+ 72us) server_negotiation.cc:407] Connection header received
0806 00:58:28.941880 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:28.941881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:28.941889 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:28.941944 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:28.942180 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.942725 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.944512 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.944887 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.944925 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.944933 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:28.945140 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 00:58:28.945175 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 00:58:28.945272 361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.941481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:41909 (local address 127.23.242.1:46367)
0806 00:58:28.941663 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:28.941679 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:28.941794 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:28.942014 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:28.942015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:28.942016 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:28.942077 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.942078 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.942817 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.942819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.944400 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.944401 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.944980 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.944981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.945045 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.945049 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 00:58:28.945134 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 00:58:28.945176 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":40,"thread_start_us":89,"threads_started":1}
I20260806 00:58:28.945760 32568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af590ebaadfa4d6c8d65037db211fe6a" candidate_uuid: "51abc5f62ed248a89acb552552d25cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6065afec92049beb971daffd555d27e" is_pre_election: true
I20260806 00:58:28.945875 32568 raft_consensus.cc:2454] T af590ebaadfa4d6c8d65037db211fe6a P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51abc5f62ed248a89acb552552d25cf5 in term 0.
I20260806 00:58:28.946058 314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.941793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:52813 (local address 127.23.242.3:36941)
0806 00:58:28.941836 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:28.941838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:28.941848 (+ 10us) server_negotiation.cc:407] Connection header received
0806 00:58:28.941878 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:28.941878 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:28.941888 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:28.941948 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:28.942139 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.942876 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.944540 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.945543 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.945737 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.945755 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:28.945943 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 00:58:28.945994 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 00:58:28.946074 32357 leader_election.cc:302] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51abc5f62ed248a89acb552552d25cf5, d6065afec92049beb971daffd555d27e; no voters:
I20260806 00:58:28.946203 328 raft_consensus.cc:2781] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 00:58:28.946257 328 raft_consensus.cc:482] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 00:58:28.946290 328 raft_consensus.cc:3037] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:28.946478 362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.941661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:36941 (local address 127.23.242.1:52813)
0806 00:58:28.941737 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:28.941746 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:28.941794 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:28.941972 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:28.941973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:28.941976 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:28.942062 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.942064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.942958 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.942958 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.944396 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.944400 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.945700 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.945703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.945828 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.945833 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 00:58:28.946145 (+ 312us) client_negotiation.cc:240] Negotiation successful
0806 00:58:28.946196 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":2,"thread_start_us":35,"threads_started":1}
I20260806 00:58:28.947132 328 raft_consensus.cc:504] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } }
I20260806 00:58:28.947244 328 leader_election.cc:288] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5 [CANDIDATE]: Term 1 election: Requested vote from peers d6065afec92049beb971daffd555d27e (127.23.242.2:41909), 4ee9f830e89d43f38f5060bf8d613b89 (127.23.242.3:36941)
I20260806 00:58:28.947528 32568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af590ebaadfa4d6c8d65037db211fe6a" candidate_uuid: "51abc5f62ed248a89acb552552d25cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6065afec92049beb971daffd555d27e"
I20260806 00:58:28.947563 32712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af590ebaadfa4d6c8d65037db211fe6a" candidate_uuid: "51abc5f62ed248a89acb552552d25cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee9f830e89d43f38f5060bf8d613b89" is_pre_election: true
I20260806 00:58:28.947642 32568 raft_consensus.cc:3037] T af590ebaadfa4d6c8d65037db211fe6a P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:28.947672 32712 raft_consensus.cc:2454] T af590ebaadfa4d6c8d65037db211fe6a P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51abc5f62ed248a89acb552552d25cf5 in term 0.
I20260806 00:58:28.947641 32711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af590ebaadfa4d6c8d65037db211fe6a" candidate_uuid: "51abc5f62ed248a89acb552552d25cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee9f830e89d43f38f5060bf8d613b89"
I20260806 00:58:28.947722 32711 raft_consensus.cc:3037] T af590ebaadfa4d6c8d65037db211fe6a P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:28.948525 32568 raft_consensus.cc:2454] T af590ebaadfa4d6c8d65037db211fe6a P d6065afec92049beb971daffd555d27e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51abc5f62ed248a89acb552552d25cf5 in term 1.
I20260806 00:58:28.948689 32711 raft_consensus.cc:2454] T af590ebaadfa4d6c8d65037db211fe6a P 4ee9f830e89d43f38f5060bf8d613b89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51abc5f62ed248a89acb552552d25cf5 in term 1.
I20260806 00:58:28.948794 32357 leader_election.cc:302] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51abc5f62ed248a89acb552552d25cf5, d6065afec92049beb971daffd555d27e; no voters:
I20260806 00:58:28.948917 328 raft_consensus.cc:2781] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:28.949067 328 raft_consensus.cc:686] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5 [term 1 LEADER]: Becoming Leader. State: Replica: 51abc5f62ed248a89acb552552d25cf5, State: Running, Role: LEADER
I20260806 00:58:28.949184 328 consensus_queue.cc:237] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } }
I20260806 00:58:28.949654 363 tablet_replica.cc:439] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5: TxnStatusTablet state changed. Reason: New leader 51abc5f62ed248a89acb552552d25cf5. Latest consensus state: current_term: 1 leader_uuid: "51abc5f62ed248a89acb552552d25cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } }
I20260806 00:58:28.949815 363 tablet_replica.cc:442] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5: This TxnStatusTablet replica's current role is: LEADER
I20260806 00:58:28.950049 32265 catalog_manager.cc:5495] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51abc5f62ed248a89acb552552d25cf5 (127.23.242.1). New cstate: current_term: 1 leader_uuid: "51abc5f62ed248a89acb552552d25cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } health_report { overall_health: HEALTHY } } }
I20260806 00:58:28.950392 365 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 00:58:28.950516 365 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 00:58:28.960071 318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.956883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41312 (local address 127.23.242.1:44129)
0806 00:58:28.956923 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:28.956925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:28.956961 (+ 36us) server_negotiation.cc:407] Connection header received
0806 00:58:28.957064 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:28.957065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:28.957082 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:28.957133 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 00:58:28.957293 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.957765 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.958901 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.959115 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.959175 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.959437 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 00:58:28.959629 (+ 192us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 00:58:28.959751 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:28.959969 (+ 218us) server_negotiation.cc:299] Negotiation successful
0806 00:58:28.959995 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 00:58:28.960196 357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.956824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:44129 (local address 127.0.0.1:41312)
0806 00:58:28.956911 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:28.956922 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:28.956981 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:28.957154 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:28.957155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:28.957157 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 00:58:28.957210 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.957212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.957851 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.957852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.958795 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.958799 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.959200 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.959201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.959280 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.959298 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 00:58:28.959815 (+ 517us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 00:58:28.959828 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 00:58:28.959941 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 00:58:28.960033 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 00:58:28.961521 32711 raft_consensus.cc:1261] T af590ebaadfa4d6c8d65037db211fe6a P 4ee9f830e89d43f38f5060bf8d613b89 [term 1 FOLLOWER]: Refusing update from remote peer 51abc5f62ed248a89acb552552d25cf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:28.961658 32568 raft_consensus.cc:1261] T af590ebaadfa4d6c8d65037db211fe6a P d6065afec92049beb971daffd555d27e [term 1 FOLLOWER]: Refusing update from remote peer 51abc5f62ed248a89acb552552d25cf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:28.961835 328 consensus_queue.cc:1035] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:28.961961 363 consensus_queue.cc:1035] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:28.962776 322 tablet_replica.cc:439] T af590ebaadfa4d6c8d65037db211fe6a P 4ee9f830e89d43f38f5060bf8d613b89: TxnStatusTablet state changed. Reason: New leader 51abc5f62ed248a89acb552552d25cf5. Latest consensus state: current_term: 1 leader_uuid: "51abc5f62ed248a89acb552552d25cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } }
I20260806 00:58:28.962906 322 tablet_replica.cc:442] T af590ebaadfa4d6c8d65037db211fe6a P 4ee9f830e89d43f38f5060bf8d613b89: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 00:58:28.962872 338 tablet_replica.cc:439] T af590ebaadfa4d6c8d65037db211fe6a P d6065afec92049beb971daffd555d27e: TxnStatusTablet state changed. Reason: New leader 51abc5f62ed248a89acb552552d25cf5. Latest consensus state: current_term: 1 leader_uuid: "51abc5f62ed248a89acb552552d25cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } }
I20260806 00:58:28.962971 338 tablet_replica.cc:442] T af590ebaadfa4d6c8d65037db211fe6a P d6065afec92049beb971daffd555d27e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 00:58:28.963486 328 tablet_replica.cc:439] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "51abc5f62ed248a89acb552552d25cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } }
I20260806 00:58:28.963574 328 tablet_replica.cc:442] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5: This TxnStatusTablet replica's current role is: LEADER
I20260806 00:58:28.963677 328 tablet_replica.cc:439] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "51abc5f62ed248a89acb552552d25cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } }
I20260806 00:58:28.963727 328 tablet_replica.cc:442] T af590ebaadfa4d6c8d65037db211fe6a P 51abc5f62ed248a89acb552552d25cf5: This TxnStatusTablet replica's current role is: LEADER
I20260806 00:58:28.963807 322 tablet_replica.cc:439] T af590ebaadfa4d6c8d65037db211fe6a P 4ee9f830e89d43f38f5060bf8d613b89: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "51abc5f62ed248a89acb552552d25cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } }
I20260806 00:58:28.963881 322 tablet_replica.cc:442] T af590ebaadfa4d6c8d65037db211fe6a P 4ee9f830e89d43f38f5060bf8d613b89: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 00:58:28.964329 333 mvcc.cc:205] Tried to move back new op lower bound from 7315365515104145408 to 7315365515056676864. Current Snapshot: MvccSnapshot[applied={T|T < 7315365515104145408}]
I20260806 00:58:28.964417 338 tablet_replica.cc:439] T af590ebaadfa4d6c8d65037db211fe6a P d6065afec92049beb971daffd555d27e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "51abc5f62ed248a89acb552552d25cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } } peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } }
I20260806 00:58:28.964506 338 tablet_replica.cc:442] T af590ebaadfa4d6c8d65037db211fe6a P d6065afec92049beb971daffd555d27e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 00:58:28.968930 314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.965646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40530 (local address 127.23.242.3:36941)
0806 00:58:28.965688 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:28.965690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:28.965700 (+ 10us) server_negotiation.cc:407] Connection header received
0806 00:58:28.965786 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:28.965787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:28.965811 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:28.965886 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 00:58:28.965998 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.966901 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.967903 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.968227 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.968308 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.968488 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 00:58:28.968626 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 00:58:28.968765 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:28.968792 (+ 27us) server_negotiation.cc:299] Negotiation successful
0806 00:58:28.968816 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 00:58:28.974937 318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.971867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:41915 (local address 127.23.242.1:44129)
0806 00:58:28.971925 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:28.971927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:28.971935 (+ 8us) server_negotiation.cc:407] Connection header received
0806 00:58:28.971957 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:28.971958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:28.971966 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:28.972014 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 00:58:28.972217 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.972781 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.974057 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.974266 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.974331 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.974578 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 00:58:28.974686 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 00:58:28.974763 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:28.974851 (+ 88us) server_negotiation.cc:299] Negotiation successful
0806 00:58:28.974873 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 00:58:28.974982 32764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.971647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:44129 (local address 127.23.242.3:41915)
0806 00:58:28.971727 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:28.971739 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:28.971783 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:28.972039 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:28.972040 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:28.972042 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 00:58:28.972109 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.972110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.972930 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.972932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.973942 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.973945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.974360 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.974361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.974437 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.974455 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 00:58:28.974778 (+ 323us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 00:58:28.974780 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 00:58:28.974839 (+ 59us) client_negotiation.cc:240] Negotiation successful
0806 00:58:28.974890 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260806 00:58:28.977579 32568 raft_consensus.cc:1261] T 1096d16cd8e34d8bac856204dd068144 P d6065afec92049beb971daffd555d27e [term 1 FOLLOWER]: Refusing update from remote peer 4ee9f830e89d43f38f5060bf8d613b89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:28.977579 32425 raft_consensus.cc:1261] T 1096d16cd8e34d8bac856204dd068144 P 51abc5f62ed248a89acb552552d25cf5 [term 1 FOLLOWER]: Refusing update from remote peer 4ee9f830e89d43f38f5060bf8d613b89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:28.977836 322 consensus_queue.cc:1035] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:28.978009 350 consensus_queue.cc:1035] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:28.987099 314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.983919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:45419 (local address 127.23.242.3:36941)
0806 00:58:28.983961 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:28.983963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:28.983992 (+ 29us) server_negotiation.cc:407] Connection header received
0806 00:58:28.984021 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:28.984022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:28.984030 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:28.984067 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 00:58:28.984247 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.984731 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.986209 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.986437 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.986520 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.986801 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 00:58:28.986844 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 00:58:28.986916 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:28.986993 (+ 77us) server_negotiation.cc:299] Negotiation successful
0806 00:58:28.987014 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 00:58:28.987157 376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:28.983731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:36941 (local address 127.23.242.1:45419)
0806 00:58:28.983892 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:28.983906 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:28.983956 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:28.984070 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:28.984071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:28.984072 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 00:58:28.984144 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.984146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.984816 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.984817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.986035 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:28.986038 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:28.986541 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:28.986543 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:28.986626 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:28.986646 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 00:58:28.986932 (+ 286us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 00:58:28.986934 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 00:58:28.986989 (+ 55us) client_negotiation.cc:240] Negotiation successful
0806 00:58:28.987036 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260806 00:58:28.996549 32265 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:58:28.996693 32265 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:58:28.997325 32265 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6398079945c47f6bc6b7163ae969179: Sending DeleteTablet for 3 replicas of tablet 1096d16cd8e34d8bac856204dd068144
I20260806 00:58:28.997586 32405 tablet_service.cc:1514] Processing DeleteTablet for tablet 1096d16cd8e34d8bac856204dd068144 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:28 UTC) from {username='oryx', principal='oryx/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:41310
I20260806 00:58:28.997679 32548 tablet_service.cc:1514] Processing DeleteTablet for tablet 1096d16cd8e34d8bac856204dd068144 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:28 UTC) from {username='oryx', principal='oryx/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:58698
I20260806 00:58:28.997772 344 tablet_replica.cc:330] T 1096d16cd8e34d8bac856204dd068144 P 51abc5f62ed248a89acb552552d25cf5: stopping tablet replica
I20260806 00:58:28.997834 344 raft_consensus.cc:2229] T 1096d16cd8e34d8bac856204dd068144 P 51abc5f62ed248a89acb552552d25cf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:28.997879 344 raft_consensus.cc:2258] T 1096d16cd8e34d8bac856204dd068144 P 51abc5f62ed248a89acb552552d25cf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:28.997951 345 tablet_replica.cc:330] T 1096d16cd8e34d8bac856204dd068144 P d6065afec92049beb971daffd555d27e: stopping tablet replica
I20260806 00:58:28.998034 345 raft_consensus.cc:2229] T 1096d16cd8e34d8bac856204dd068144 P d6065afec92049beb971daffd555d27e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:28.998076 345 raft_consensus.cc:2258] T 1096d16cd8e34d8bac856204dd068144 P d6065afec92049beb971daffd555d27e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:28.998376 32692 tablet_service.cc:1514] Processing DeleteTablet for tablet 1096d16cd8e34d8bac856204dd068144 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:28 UTC) from {username='oryx', principal='oryx/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:40526
I20260806 00:58:28.998492 343 tablet_replica.cc:330] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89: stopping tablet replica
I20260806 00:58:28.998474 345 ts_tablet_manager.cc:1905] T 1096d16cd8e34d8bac856204dd068144 P d6065afec92049beb971daffd555d27e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:28.998565 343 raft_consensus.cc:2229] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89 [term 1 LEADER]: Raft consensus shutting down.
I20260806 00:58:28.998807 343 raft_consensus.cc:2258] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:28.999073 344 ts_tablet_manager.cc:1905] T 1096d16cd8e34d8bac856204dd068144 P 51abc5f62ed248a89acb552552d25cf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:28.999119 343 ts_tablet_manager.cc:1905] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:29.000875 344 ts_tablet_manager.cc:1918] T 1096d16cd8e34d8bac856204dd068144 P 51abc5f62ed248a89acb552552d25cf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 00:58:29.000878 343 ts_tablet_manager.cc:1918] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 00:58:29.000881 345 ts_tablet_manager.cc:1918] T 1096d16cd8e34d8bac856204dd068144 P d6065afec92049beb971daffd555d27e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 00:58:29.000932 343 log.cc:1196] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-2/wal/wals/1096d16cd8e34d8bac856204dd068144
I20260806 00:58:29.000932 344 log.cc:1196] T 1096d16cd8e34d8bac856204dd068144 P 51abc5f62ed248a89acb552552d25cf5: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-0/wal/wals/1096d16cd8e34d8bac856204dd068144
I20260806 00:58:29.000941 345 log.cc:1196] T 1096d16cd8e34d8bac856204dd068144 P d6065afec92049beb971daffd555d27e: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-1/wal/wals/1096d16cd8e34d8bac856204dd068144
I20260806 00:58:29.001171 344 ts_tablet_manager.cc:1939] T 1096d16cd8e34d8bac856204dd068144 P 51abc5f62ed248a89acb552552d25cf5: Deleting consensus metadata
I20260806 00:58:29.001171 343 ts_tablet_manager.cc:1939] T 1096d16cd8e34d8bac856204dd068144 P 4ee9f830e89d43f38f5060bf8d613b89: Deleting consensus metadata
I20260806 00:58:29.001194 345 ts_tablet_manager.cc:1939] T 1096d16cd8e34d8bac856204dd068144 P d6065afec92049beb971daffd555d27e: Deleting consensus metadata
I20260806 00:58:29.001883 32250 catalog_manager.cc:4841] TS 51abc5f62ed248a89acb552552d25cf5 (127.23.242.1:44129): tablet 1096d16cd8e34d8bac856204dd068144 (table test-table [id=22aac58c04b94741b0ccb43248c37fd9]) successfully deleted
I20260806 00:58:29.001963 32250 catalog_manager.cc:4841] TS d6065afec92049beb971daffd555d27e (127.23.242.2:41909): tablet 1096d16cd8e34d8bac856204dd068144 (table test-table [id=22aac58c04b94741b0ccb43248c37fd9]) successfully deleted
I20260806 00:58:29.004815 32250 catalog_manager.cc:4841] TS 4ee9f830e89d43f38f5060bf8d613b89 (127.23.242.3:36941): tablet 1096d16cd8e34d8bac856204dd068144 (table test-table [id=22aac58c04b94741b0ccb43248c37fd9]) successfully deleted
Aug 06 00:58:29 dist-test-slave-dhph krb5kdc[32219](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 00:58:29 dist-test-slave-dhph krb5kdc[32219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:29 dist-test-slave-dhph krb5kdc[32219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 00:58:29.033162 32316 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:29.027574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34530 (local address 127.23.242.62:39515)
0806 00:58:29.027633 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:29.027635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:29.027778 (+ 143us) server_negotiation.cc:407] Connection header received
0806 00:58:29.027871 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:29.027871 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:29.027884 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:29.027932 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:29.028531 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:29.029036 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:29.029814 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:29.030006 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:29.030096 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:29.033094 (+ 2998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34530: BlockingRecv error: recv got EOF from 127.0.0.1:34530 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Aug 06 00:58:29 dist-test-slave-dhph krb5kdc[32219](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM
I20260806 00:58:29.042879 32316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:29.034255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34542 (local address 127.23.242.62:39515)
0806 00:58:29.034294 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:29.034296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:29.034492 (+ 196us) server_negotiation.cc:407] Connection header received
0806 00:58:29.034597 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:29.034598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:29.034609 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:29.034659 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:29.035147 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:29.035699 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:29.036445 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:29.036669 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:29.036745 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:29.039017 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:29.039018 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:29.039019 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:29.039026 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:29.041144 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:29.041661 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:29.041662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:29.041663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:29.041749 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:29.042172 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:29.042174 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:29.042174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:29.042391 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:29.042507 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:29.042656 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 00:58:29.042760 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 00:58:29.044801 32265 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 00:58:29.044996 32265 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 00:58:29.046329 32548 tablet_service.cc:1467] Processing CreateTablet for tablet 03b18bc92fc8425dab0a0e0603b2caa5 (DEFAULT_TABLE table=test-table [id=7e45e2eb09574e13bafdafd57df6addf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:29.046427 32692 tablet_service.cc:1467] Processing CreateTablet for tablet 03b18bc92fc8425dab0a0e0603b2caa5 (DEFAULT_TABLE table=test-table [id=7e45e2eb09574e13bafdafd57df6addf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:29.046483 32548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b18bc92fc8425dab0a0e0603b2caa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:29.046540 32692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b18bc92fc8425dab0a0e0603b2caa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:29.047925 319 tablet_bootstrap.cc:492] T 03b18bc92fc8425dab0a0e0603b2caa5 P 4ee9f830e89d43f38f5060bf8d613b89: Bootstrap starting.
I20260806 00:58:29.047959 32405 tablet_service.cc:1467] Processing CreateTablet for tablet 03b18bc92fc8425dab0a0e0603b2caa5 (DEFAULT_TABLE table=test-table [id=7e45e2eb09574e13bafdafd57df6addf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:29.048085 32405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03b18bc92fc8425dab0a0e0603b2caa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:29.048353 319 tablet_bootstrap.cc:654] T 03b18bc92fc8425dab0a0e0603b2caa5 P 4ee9f830e89d43f38f5060bf8d613b89: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:29.049754 324 tablet_bootstrap.cc:492] T 03b18bc92fc8425dab0a0e0603b2caa5 P 51abc5f62ed248a89acb552552d25cf5: Bootstrap starting.
I20260806 00:58:29.050177 324 tablet_bootstrap.cc:654] T 03b18bc92fc8425dab0a0e0603b2caa5 P 51abc5f62ed248a89acb552552d25cf5: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:29.050488 321 tablet_bootstrap.cc:492] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e: Bootstrap starting.
I20260806 00:58:29.050869 321 tablet_bootstrap.cc:654] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:29.051177 324 tablet_bootstrap.cc:492] T 03b18bc92fc8425dab0a0e0603b2caa5 P 51abc5f62ed248a89acb552552d25cf5: No bootstrap required, opened a new log
I20260806 00:58:29.051231 324 ts_tablet_manager.cc:1397] T 03b18bc92fc8425dab0a0e0603b2caa5 P 51abc5f62ed248a89acb552552d25cf5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:29.051414 324 raft_consensus.cc:348] T 03b18bc92fc8425dab0a0e0603b2caa5 P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } }
I20260806 00:58:29.051510 324 raft_consensus.cc:374] T 03b18bc92fc8425dab0a0e0603b2caa5 P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:29.051540 324 raft_consensus.cc:729] T 03b18bc92fc8425dab0a0e0603b2caa5 P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51abc5f62ed248a89acb552552d25cf5, State: Initialized, Role: FOLLOWER
I20260806 00:58:29.051681 324 consensus_queue.cc:260] T 03b18bc92fc8425dab0a0e0603b2caa5 P 51abc5f62ed248a89acb552552d25cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } }
I20260806 00:58:29.051834 324 ts_tablet_manager.cc:1428] T 03b18bc92fc8425dab0a0e0603b2caa5 P 51abc5f62ed248a89acb552552d25cf5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 00:58:29.052139 319 tablet_bootstrap.cc:492] T 03b18bc92fc8425dab0a0e0603b2caa5 P 4ee9f830e89d43f38f5060bf8d613b89: No bootstrap required, opened a new log
I20260806 00:58:29.052182 319 ts_tablet_manager.cc:1397] T 03b18bc92fc8425dab0a0e0603b2caa5 P 4ee9f830e89d43f38f5060bf8d613b89: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260806 00:58:29.052310 319 raft_consensus.cc:348] T 03b18bc92fc8425dab0a0e0603b2caa5 P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } }
I20260806 00:58:29.052371 319 raft_consensus.cc:374] T 03b18bc92fc8425dab0a0e0603b2caa5 P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:29.052385 319 raft_consensus.cc:729] T 03b18bc92fc8425dab0a0e0603b2caa5 P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee9f830e89d43f38f5060bf8d613b89, State: Initialized, Role: FOLLOWER
I20260806 00:58:29.052457 319 consensus_queue.cc:260] T 03b18bc92fc8425dab0a0e0603b2caa5 P 4ee9f830e89d43f38f5060bf8d613b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } }
I20260806 00:58:29.052559 319 ts_tablet_manager.cc:1428] T 03b18bc92fc8425dab0a0e0603b2caa5 P 4ee9f830e89d43f38f5060bf8d613b89: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:29.055146 321 tablet_bootstrap.cc:492] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e: No bootstrap required, opened a new log
I20260806 00:58:29.055233 321 ts_tablet_manager.cc:1397] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260806 00:58:29.055383 321 raft_consensus.cc:348] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } }
I20260806 00:58:29.055500 321 raft_consensus.cc:374] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:29.055527 321 raft_consensus.cc:729] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6065afec92049beb971daffd555d27e, State: Initialized, Role: FOLLOWER
I20260806 00:58:29.055574 321 consensus_queue.cc:260] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } }
I20260806 00:58:29.055677 321 ts_tablet_manager.cc:1428] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260806 00:58:29.172933 338 raft_consensus.cc:482] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 00:58:29.173058 338 raft_consensus.cc:504] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } }
I20260806 00:58:29.173277 338 leader_election.cc:288] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51abc5f62ed248a89acb552552d25cf5 (127.23.242.1:44129), 4ee9f830e89d43f38f5060bf8d613b89 (127.23.242.3:36941)
I20260806 00:58:29.173710 32425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b18bc92fc8425dab0a0e0603b2caa5" candidate_uuid: "d6065afec92049beb971daffd555d27e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51abc5f62ed248a89acb552552d25cf5" is_pre_election: true
I20260806 00:58:29.173827 32425 raft_consensus.cc:2454] T 03b18bc92fc8425dab0a0e0603b2caa5 P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6065afec92049beb971daffd555d27e in term 0.
I20260806 00:58:29.173753 32711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b18bc92fc8425dab0a0e0603b2caa5" candidate_uuid: "d6065afec92049beb971daffd555d27e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee9f830e89d43f38f5060bf8d613b89" is_pre_election: true
I20260806 00:58:29.173878 32711 raft_consensus.cc:2454] T 03b18bc92fc8425dab0a0e0603b2caa5 P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6065afec92049beb971daffd555d27e in term 0.
I20260806 00:58:29.174021 32499 leader_election.cc:302] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51abc5f62ed248a89acb552552d25cf5, d6065afec92049beb971daffd555d27e; no voters:
I20260806 00:58:29.174156 338 raft_consensus.cc:2781] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 00:58:29.174201 338 raft_consensus.cc:482] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 00:58:29.174234 338 raft_consensus.cc:3037] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:29.174969 338 raft_consensus.cc:504] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } }
I20260806 00:58:29.175123 338 leader_election.cc:288] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e [CANDIDATE]: Term 1 election: Requested vote from peers 51abc5f62ed248a89acb552552d25cf5 (127.23.242.1:44129), 4ee9f830e89d43f38f5060bf8d613b89 (127.23.242.3:36941)
I20260806 00:58:29.175309 32425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b18bc92fc8425dab0a0e0603b2caa5" candidate_uuid: "d6065afec92049beb971daffd555d27e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51abc5f62ed248a89acb552552d25cf5"
I20260806 00:58:29.175390 32425 raft_consensus.cc:3037] T 03b18bc92fc8425dab0a0e0603b2caa5 P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:29.175344 32711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03b18bc92fc8425dab0a0e0603b2caa5" candidate_uuid: "d6065afec92049beb971daffd555d27e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee9f830e89d43f38f5060bf8d613b89"
I20260806 00:58:29.175450 32711 raft_consensus.cc:3037] T 03b18bc92fc8425dab0a0e0603b2caa5 P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:29.175995 32425 raft_consensus.cc:2454] T 03b18bc92fc8425dab0a0e0603b2caa5 P 51abc5f62ed248a89acb552552d25cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6065afec92049beb971daffd555d27e in term 1.
I20260806 00:58:29.176198 32499 leader_election.cc:302] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51abc5f62ed248a89acb552552d25cf5, d6065afec92049beb971daffd555d27e; no voters:
I20260806 00:58:29.176272 32711 raft_consensus.cc:2454] T 03b18bc92fc8425dab0a0e0603b2caa5 P 4ee9f830e89d43f38f5060bf8d613b89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6065afec92049beb971daffd555d27e in term 1.
I20260806 00:58:29.176296 338 raft_consensus.cc:2781] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:29.176344 338 raft_consensus.cc:686] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e [term 1 LEADER]: Becoming Leader. State: Replica: d6065afec92049beb971daffd555d27e, State: Running, Role: LEADER
I20260806 00:58:29.176400 338 consensus_queue.cc:237] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } }
I20260806 00:58:29.177080 32266 catalog_manager.cc:5495] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e reported cstate change: term changed from 0 to 1, leader changed from <none> to d6065afec92049beb971daffd555d27e (127.23.242.2). New cstate: current_term: 1 leader_uuid: "d6065afec92049beb971daffd555d27e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } health_report { overall_health: UNKNOWN } } }
I20260806 00:58:29.202104 316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:29.199535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58712 (local address 127.23.242.2:41909)
0806 00:58:29.199652 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:29.199654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:29.199662 (+ 8us) server_negotiation.cc:407] Connection header received
0806 00:58:29.199686 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:29.199687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:29.199695 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:29.199746 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 00:58:29.199942 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:29.200433 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:29.201271 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:29.201464 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:29.201528 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:29.201725 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 00:58:29.201754 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 00:58:29.201800 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:29.201946 (+ 146us) server_negotiation.cc:299] Negotiation successful
0806 00:58:29.201992 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 00:58:29.203068 32711 raft_consensus.cc:1261] T 03b18bc92fc8425dab0a0e0603b2caa5 P 4ee9f830e89d43f38f5060bf8d613b89 [term 1 FOLLOWER]: Refusing update from remote peer d6065afec92049beb971daffd555d27e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:29.203107 32425 raft_consensus.cc:1261] T 03b18bc92fc8425dab0a0e0603b2caa5 P 51abc5f62ed248a89acb552552d25cf5 [term 1 FOLLOWER]: Refusing update from remote peer d6065afec92049beb971daffd555d27e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:29.203377 338 consensus_queue.cc:1035] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:29.203533 330 consensus_queue.cc:1035] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:29.206715 32265 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:58:29.206833 32265 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:58:29.207921 32265 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6398079945c47f6bc6b7163ae969179: Sending DeleteTablet for 3 replicas of tablet 03b18bc92fc8425dab0a0e0603b2caa5
I20260806 00:58:29.208304 32405 tablet_service.cc:1514] Processing DeleteTablet for tablet 03b18bc92fc8425dab0a0e0603b2caa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:29 UTC) from {username='oryx', principal='oryx/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:41310
I20260806 00:58:29.208333 32548 tablet_service.cc:1514] Processing DeleteTablet for tablet 03b18bc92fc8425dab0a0e0603b2caa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:29 UTC) from {username='oryx', principal='oryx/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:58698
I20260806 00:58:29.208422 345 tablet_replica.cc:330] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e: stopping tablet replica
I20260806 00:58:29.208446 344 tablet_replica.cc:330] T 03b18bc92fc8425dab0a0e0603b2caa5 P 51abc5f62ed248a89acb552552d25cf5: stopping tablet replica
I20260806 00:58:29.208477 345 raft_consensus.cc:2229] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e [term 1 LEADER]: Raft consensus shutting down.
I20260806 00:58:29.208483 32692 tablet_service.cc:1514] Processing DeleteTablet for tablet 03b18bc92fc8425dab0a0e0603b2caa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:29 UTC) from {username='oryx', principal='oryx/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:40526
I20260806 00:58:29.208531 344 raft_consensus.cc:2229] T 03b18bc92fc8425dab0a0e0603b2caa5 P 51abc5f62ed248a89acb552552d25cf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:29.208544 345 raft_consensus.cc:2258] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:29.208563 343 tablet_replica.cc:330] T 03b18bc92fc8425dab0a0e0603b2caa5 P 4ee9f830e89d43f38f5060bf8d613b89: stopping tablet replica
I20260806 00:58:29.208601 344 raft_consensus.cc:2258] T 03b18bc92fc8425dab0a0e0603b2caa5 P 51abc5f62ed248a89acb552552d25cf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:29.208606 343 raft_consensus.cc:2229] T 03b18bc92fc8425dab0a0e0603b2caa5 P 4ee9f830e89d43f38f5060bf8d613b89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:29.208652 343 raft_consensus.cc:2258] T 03b18bc92fc8425dab0a0e0603b2caa5 P 4ee9f830e89d43f38f5060bf8d613b89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:29.208578 32265 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 00:58:29.208778 32265 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 00:58:29.208940 344 ts_tablet_manager.cc:1905] T 03b18bc92fc8425dab0a0e0603b2caa5 P 51abc5f62ed248a89acb552552d25cf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:29.208973 343 ts_tablet_manager.cc:1905] T 03b18bc92fc8425dab0a0e0603b2caa5 P 4ee9f830e89d43f38f5060bf8d613b89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:29.209134 345 ts_tablet_manager.cc:1905] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:29.210234 32692 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0d46c32db34159a31b60c38e73c200 (DEFAULT_TABLE table=test-table [id=adf9e4a6f3f642629cdeca92c8de02d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:29.210353 32692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f0d46c32db34159a31b60c38e73c200. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:29.211226 344 ts_tablet_manager.cc:1918] T 03b18bc92fc8425dab0a0e0603b2caa5 P 51abc5f62ed248a89acb552552d25cf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 00:58:29.211308 344 log.cc:1196] T 03b18bc92fc8425dab0a0e0603b2caa5 P 51abc5f62ed248a89acb552552d25cf5: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-0/wal/wals/03b18bc92fc8425dab0a0e0603b2caa5
I20260806 00:58:29.211464 345 ts_tablet_manager.cc:1918] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 00:58:29.211483 343 ts_tablet_manager.cc:1918] T 03b18bc92fc8425dab0a0e0603b2caa5 P 4ee9f830e89d43f38f5060bf8d613b89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 00:58:29.211539 345 log.cc:1196] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-1/wal/wals/03b18bc92fc8425dab0a0e0603b2caa5
I20260806 00:58:29.211557 344 ts_tablet_manager.cc:1939] T 03b18bc92fc8425dab0a0e0603b2caa5 P 51abc5f62ed248a89acb552552d25cf5: Deleting consensus metadata
I20260806 00:58:29.211613 343 log.cc:1196] T 03b18bc92fc8425dab0a0e0603b2caa5 P 4ee9f830e89d43f38f5060bf8d613b89: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-2/wal/wals/03b18bc92fc8425dab0a0e0603b2caa5
I20260806 00:58:29.211787 345 ts_tablet_manager.cc:1939] T 03b18bc92fc8425dab0a0e0603b2caa5 P d6065afec92049beb971daffd555d27e: Deleting consensus metadata
I20260806 00:58:29.211869 343 ts_tablet_manager.cc:1939] T 03b18bc92fc8425dab0a0e0603b2caa5 P 4ee9f830e89d43f38f5060bf8d613b89: Deleting consensus metadata
I20260806 00:58:29.212046 319 tablet_bootstrap.cc:492] T 9f0d46c32db34159a31b60c38e73c200 P 4ee9f830e89d43f38f5060bf8d613b89: Bootstrap starting.
I20260806 00:58:29.212203 32250 catalog_manager.cc:4841] TS d6065afec92049beb971daffd555d27e (127.23.242.2:41909): tablet 03b18bc92fc8425dab0a0e0603b2caa5 (table test-table [id=7e45e2eb09574e13bafdafd57df6addf]) successfully deleted
I20260806 00:58:29.212291 32250 catalog_manager.cc:4841] TS 4ee9f830e89d43f38f5060bf8d613b89 (127.23.242.3:36941): tablet 03b18bc92fc8425dab0a0e0603b2caa5 (table test-table [id=7e45e2eb09574e13bafdafd57df6addf]) successfully deleted
I20260806 00:58:29.212328 32548 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0d46c32db34159a31b60c38e73c200 (DEFAULT_TABLE table=test-table [id=adf9e4a6f3f642629cdeca92c8de02d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:29.212424 32548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f0d46c32db34159a31b60c38e73c200. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:29.212431 319 tablet_bootstrap.cc:654] T 9f0d46c32db34159a31b60c38e73c200 P 4ee9f830e89d43f38f5060bf8d613b89: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:29.212630 32250 catalog_manager.cc:4841] TS 51abc5f62ed248a89acb552552d25cf5 (127.23.242.1:44129): tablet 03b18bc92fc8425dab0a0e0603b2caa5 (table test-table [id=7e45e2eb09574e13bafdafd57df6addf]) successfully deleted
I20260806 00:58:29.212623 32405 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0d46c32db34159a31b60c38e73c200 (DEFAULT_TABLE table=test-table [id=adf9e4a6f3f642629cdeca92c8de02d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:29.212728 32405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f0d46c32db34159a31b60c38e73c200. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:29.213056 319 tablet_bootstrap.cc:492] T 9f0d46c32db34159a31b60c38e73c200 P 4ee9f830e89d43f38f5060bf8d613b89: No bootstrap required, opened a new log
I20260806 00:58:29.213099 319 ts_tablet_manager.cc:1397] T 9f0d46c32db34159a31b60c38e73c200 P 4ee9f830e89d43f38f5060bf8d613b89: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:29.213313 319 raft_consensus.cc:348] T 9f0d46c32db34159a31b60c38e73c200 P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } }
I20260806 00:58:29.213409 319 raft_consensus.cc:374] T 9f0d46c32db34159a31b60c38e73c200 P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:29.213434 319 raft_consensus.cc:729] T 9f0d46c32db34159a31b60c38e73c200 P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee9f830e89d43f38f5060bf8d613b89, State: Initialized, Role: FOLLOWER
I20260806 00:58:29.213456 321 tablet_bootstrap.cc:492] T 9f0d46c32db34159a31b60c38e73c200 P d6065afec92049beb971daffd555d27e: Bootstrap starting.
I20260806 00:58:29.213481 319 consensus_queue.cc:260] T 9f0d46c32db34159a31b60c38e73c200 P 4ee9f830e89d43f38f5060bf8d613b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } }
I20260806 00:58:29.213579 319 ts_tablet_manager.cc:1428] T 9f0d46c32db34159a31b60c38e73c200 P 4ee9f830e89d43f38f5060bf8d613b89: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260806 00:58:29.213841 321 tablet_bootstrap.cc:654] T 9f0d46c32db34159a31b60c38e73c200 P d6065afec92049beb971daffd555d27e: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:29.214352 324 tablet_bootstrap.cc:492] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5: Bootstrap starting.
I20260806 00:58:29.214715 324 tablet_bootstrap.cc:654] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:29.215085 321 tablet_bootstrap.cc:492] T 9f0d46c32db34159a31b60c38e73c200 P d6065afec92049beb971daffd555d27e: No bootstrap required, opened a new log
I20260806 00:58:29.215152 321 ts_tablet_manager.cc:1397] T 9f0d46c32db34159a31b60c38e73c200 P d6065afec92049beb971daffd555d27e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 00:58:29.215260 321 raft_consensus.cc:348] T 9f0d46c32db34159a31b60c38e73c200 P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } }
I20260806 00:58:29.215355 321 raft_consensus.cc:374] T 9f0d46c32db34159a31b60c38e73c200 P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:29.215384 321 raft_consensus.cc:729] T 9f0d46c32db34159a31b60c38e73c200 P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6065afec92049beb971daffd555d27e, State: Initialized, Role: FOLLOWER
I20260806 00:58:29.215426 321 consensus_queue.cc:260] T 9f0d46c32db34159a31b60c38e73c200 P d6065afec92049beb971daffd555d27e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } }
I20260806 00:58:29.215520 321 ts_tablet_manager.cc:1428] T 9f0d46c32db34159a31b60c38e73c200 P d6065afec92049beb971daffd555d27e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:29.215561 324 tablet_bootstrap.cc:492] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5: No bootstrap required, opened a new log
I20260806 00:58:29.215653 324 ts_tablet_manager.cc:1397] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:29.215785 324 raft_consensus.cc:348] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } }
I20260806 00:58:29.215869 324 raft_consensus.cc:374] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:29.215884 324 raft_consensus.cc:729] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51abc5f62ed248a89acb552552d25cf5, State: Initialized, Role: FOLLOWER
I20260806 00:58:29.215939 324 consensus_queue.cc:260] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } }
I20260806 00:58:29.216041 324 ts_tablet_manager.cc:1428] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260806 00:58:29.244254 32617 tablet.cc:2369] T 9f0d46c32db34159a31b60c38e73c200 P d6065afec92049beb971daffd555d27e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 00:58:29.333745 372 raft_consensus.cc:482] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 00:58:29.333895 372 raft_consensus.cc:504] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } }
I20260806 00:58:29.334074 372 leader_election.cc:288] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6065afec92049beb971daffd555d27e (127.23.242.2:41909), 4ee9f830e89d43f38f5060bf8d613b89 (127.23.242.3:36941)
I20260806 00:58:29.334349 32711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0d46c32db34159a31b60c38e73c200" candidate_uuid: "51abc5f62ed248a89acb552552d25cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee9f830e89d43f38f5060bf8d613b89" is_pre_election: true
I20260806 00:58:29.334445 32711 raft_consensus.cc:2454] T 9f0d46c32db34159a31b60c38e73c200 P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51abc5f62ed248a89acb552552d25cf5 in term 0.
I20260806 00:58:29.334357 32567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0d46c32db34159a31b60c38e73c200" candidate_uuid: "51abc5f62ed248a89acb552552d25cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6065afec92049beb971daffd555d27e" is_pre_election: true
I20260806 00:58:29.334493 32567 raft_consensus.cc:2454] T 9f0d46c32db34159a31b60c38e73c200 P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51abc5f62ed248a89acb552552d25cf5 in term 0.
I20260806 00:58:29.334635 32357 leader_election.cc:302] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ee9f830e89d43f38f5060bf8d613b89, 51abc5f62ed248a89acb552552d25cf5; no voters:
I20260806 00:58:29.334760 372 raft_consensus.cc:2781] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 00:58:29.334830 372 raft_consensus.cc:482] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 00:58:29.334867 372 raft_consensus.cc:3037] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:29.335722 372 raft_consensus.cc:504] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } }
I20260806 00:58:29.335894 372 leader_election.cc:288] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5 [CANDIDATE]: Term 1 election: Requested vote from peers d6065afec92049beb971daffd555d27e (127.23.242.2:41909), 4ee9f830e89d43f38f5060bf8d613b89 (127.23.242.3:36941)
I20260806 00:58:29.336066 32567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0d46c32db34159a31b60c38e73c200" candidate_uuid: "51abc5f62ed248a89acb552552d25cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6065afec92049beb971daffd555d27e"
I20260806 00:58:29.336158 32567 raft_consensus.cc:3037] T 9f0d46c32db34159a31b60c38e73c200 P d6065afec92049beb971daffd555d27e [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:29.336102 32711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f0d46c32db34159a31b60c38e73c200" candidate_uuid: "51abc5f62ed248a89acb552552d25cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee9f830e89d43f38f5060bf8d613b89"
I20260806 00:58:29.336226 32711 raft_consensus.cc:3037] T 9f0d46c32db34159a31b60c38e73c200 P 4ee9f830e89d43f38f5060bf8d613b89 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:29.336877 32711 raft_consensus.cc:2454] T 9f0d46c32db34159a31b60c38e73c200 P 4ee9f830e89d43f38f5060bf8d613b89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51abc5f62ed248a89acb552552d25cf5 in term 1.
I20260806 00:58:29.336967 32567 raft_consensus.cc:2454] T 9f0d46c32db34159a31b60c38e73c200 P d6065afec92049beb971daffd555d27e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51abc5f62ed248a89acb552552d25cf5 in term 1.
I20260806 00:58:29.337055 32357 leader_election.cc:302] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ee9f830e89d43f38f5060bf8d613b89, 51abc5f62ed248a89acb552552d25cf5; no voters:
I20260806 00:58:29.337210 372 raft_consensus.cc:2781] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:29.337294 372 raft_consensus.cc:686] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5 [term 1 LEADER]: Becoming Leader. State: Replica: 51abc5f62ed248a89acb552552d25cf5, State: Running, Role: LEADER
I20260806 00:58:29.337355 372 consensus_queue.cc:237] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } }
I20260806 00:58:29.338191 32264 catalog_manager.cc:5495] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51abc5f62ed248a89acb552552d25cf5 (127.23.242.1). New cstate: current_term: 1 leader_uuid: "51abc5f62ed248a89acb552552d25cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51abc5f62ed248a89acb552552d25cf5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 } health_report { overall_health: UNKNOWN } } }
I20260806 00:58:29.370239 318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:29.367820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41326 (local address 127.23.242.1:44129)
0806 00:58:29.367864 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:29.367866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:29.367874 (+ 8us) server_negotiation.cc:407] Connection header received
0806 00:58:29.367928 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:29.367929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:29.367938 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:29.367990 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 00:58:29.368094 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:29.368560 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:29.369477 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:29.369715 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:29.369781 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:29.369993 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 00:58:29.370024 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 00:58:29.370075 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:29.370150 (+ 75us) server_negotiation.cc:299] Negotiation successful
0806 00:58:29.370172 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 00:58:29.376399 376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:29.373404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:44129 (local address 127.23.242.1:47757)
0806 00:58:29.373497 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:29.373511 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:29.373566 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:29.373779 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:29.373780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:29.373782 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 00:58:29.373818 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:29.373820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:29.374504 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:29.374505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:29.375388 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:29.375396 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:29.375957 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:29.375958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:29.376009 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:29.376016 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 00:58:29.376244 (+ 228us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 00:58:29.376247 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 00:58:29.376293 (+ 46us) client_negotiation.cc:240] Negotiation successful
0806 00:58:29.376339 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 00:58:29.376492 318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:29.373504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:47757 (local address 127.23.242.1:44129)
0806 00:58:29.373560 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:29.373562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:29.373572 (+ 10us) server_negotiation.cc:407] Connection header received
0806 00:58:29.373632 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:29.373632 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:29.373641 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:29.373694 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 00:58:29.373917 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:29.374391 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:29.375607 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:29.375826 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:29.375904 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:29.376124 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 00:58:29.376158 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 00:58:29.376213 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:29.376366 (+ 153us) server_negotiation.cc:299] Negotiation successful
0806 00:58:29.376395 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 00:58:29.378612 32711 raft_consensus.cc:1261] T 9f0d46c32db34159a31b60c38e73c200 P 4ee9f830e89d43f38f5060bf8d613b89 [term 1 FOLLOWER]: Refusing update from remote peer 51abc5f62ed248a89acb552552d25cf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:29.378660 32567 raft_consensus.cc:1261] T 9f0d46c32db34159a31b60c38e73c200 P d6065afec92049beb971daffd555d27e [term 1 FOLLOWER]: Refusing update from remote peer 51abc5f62ed248a89acb552552d25cf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:29.378862 363 consensus_queue.cc:1035] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ee9f830e89d43f38f5060bf8d613b89" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:29.378998 328 consensus_queue.cc:1035] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6065afec92049beb971daffd555d27e" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:29.391217 32264 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:58:29.391353 32264 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:58:29.392197 32264 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6398079945c47f6bc6b7163ae969179: Sending DeleteTablet for 3 replicas of tablet 9f0d46c32db34159a31b60c38e73c200
I20260806 00:58:29.392519 32692 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0d46c32db34159a31b60c38e73c200 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:29 UTC) from {username='oryx', principal='oryx/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:40526
I20260806 00:58:29.392519 32405 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0d46c32db34159a31b60c38e73c200 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:29 UTC) from {username='oryx', principal='oryx/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:41310
I20260806 00:58:29.392616 344 tablet_replica.cc:330] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5: stopping tablet replica
I20260806 00:58:29.392625 343 tablet_replica.cc:330] T 9f0d46c32db34159a31b60c38e73c200 P 4ee9f830e89d43f38f5060bf8d613b89: stopping tablet replica
I20260806 00:58:29.392578 32548 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f0d46c32db34159a31b60c38e73c200 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:29 UTC) from {username='oryx', principal='oryx/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:58698
I20260806 00:58:29.392669 344 raft_consensus.cc:2229] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5 [term 1 LEADER]: Raft consensus shutting down.
I20260806 00:58:29.392695 343 raft_consensus.cc:2229] T 9f0d46c32db34159a31b60c38e73c200 P 4ee9f830e89d43f38f5060bf8d613b89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:29.392705 345 tablet_replica.cc:330] T 9f0d46c32db34159a31b60c38e73c200 P d6065afec92049beb971daffd555d27e: stopping tablet replica
I20260806 00:58:29.392747 343 raft_consensus.cc:2258] T 9f0d46c32db34159a31b60c38e73c200 P 4ee9f830e89d43f38f5060bf8d613b89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:29.392791 345 raft_consensus.cc:2229] T 9f0d46c32db34159a31b60c38e73c200 P d6065afec92049beb971daffd555d27e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:29.392832 345 raft_consensus.cc:2258] T 9f0d46c32db34159a31b60c38e73c200 P d6065afec92049beb971daffd555d27e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:29.392831 344 raft_consensus.cc:2258] T 9f0d46c32db34159a31b60c38e73c200 P 51abc5f62ed248a89acb552552d25cf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:29.393191 345 ts_tablet_manager.cc:1905] T 9f0d46c32db34159a31b60c38e73c200 P d6065afec92049beb971daffd555d27e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:29.393204 343 ts_tablet_manager.cc:1905] T 9f0d46c32db34159a31b60c38e73c200 P 4ee9f830e89d43f38f5060bf8d613b89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:29.393216 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 32339
I20260806 00:58:29.395262 343 ts_tablet_manager.cc:1918] T 9f0d46c32db34159a31b60c38e73c200 P 4ee9f830e89d43f38f5060bf8d613b89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 00:58:29.395329 343 log.cc:1196] T 9f0d46c32db34159a31b60c38e73c200 P 4ee9f830e89d43f38f5060bf8d613b89: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-2/wal/wals/9f0d46c32db34159a31b60c38e73c200
I20260806 00:58:29.395603 343 ts_tablet_manager.cc:1939] T 9f0d46c32db34159a31b60c38e73c200 P 4ee9f830e89d43f38f5060bf8d613b89: Deleting consensus metadata
I20260806 00:58:29.395955 345 ts_tablet_manager.cc:1918] T 9f0d46c32db34159a31b60c38e73c200 P d6065afec92049beb971daffd555d27e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 00:58:29.395994 32250 catalog_manager.cc:4841] TS 4ee9f830e89d43f38f5060bf8d613b89 (127.23.242.3:36941): tablet 9f0d46c32db34159a31b60c38e73c200 (table test-table [id=adf9e4a6f3f642629cdeca92c8de02d1]) successfully deleted
I20260806 00:58:29.396027 345 log.cc:1196] T 9f0d46c32db34159a31b60c38e73c200 P d6065afec92049beb971daffd555d27e: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785977869987768-24520-0/minicluster-data/ts-1/wal/wals/9f0d46c32db34159a31b60c38e73c200
I20260806 00:58:29.396260 345 ts_tablet_manager.cc:1939] T 9f0d46c32db34159a31b60c38e73c200 P d6065afec92049beb971daffd555d27e: Deleting consensus metadata
I20260806 00:58:29.396639 32250 catalog_manager.cc:4841] TS d6065afec92049beb971daffd555d27e (127.23.242.2:41909): tablet 9f0d46c32db34159a31b60c38e73c200 (table test-table [id=adf9e4a6f3f642629cdeca92c8de02d1]) successfully deleted
W20260806 00:58:29.402771 32250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.242.1:44129 (error 108)
I20260806 00:58:29.403059 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 32482
I20260806 00:58:29.404754 407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:29.404531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:44129 (local address 127.0.0.1:41338)
0806 00:58:29.404656 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:29.404709 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.1:44129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":84,"threads_started":1}
W20260806 00:58:29.405004 32250 catalog_manager.cc:4584] TS 51abc5f62ed248a89acb552552d25cf5 (127.23.242.1:44129): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9f0d46c32db34159a31b60c38e73c200: Network error: Client connection negotiation failed: client connection to 127.23.242.1:44129: connect: Connection refused (error 111)
I20260806 00:58:29.410934 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 32625
I20260806 00:58:29.418833 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 32232
2026-08-06T00:58:29Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2817 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 00:58:29 dist-test-slave-dhph krb5kdc[410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 00:58:29 dist-test-slave-dhph krb5kdc[410](info): set up 2 sockets
Aug 06 00:58:29 dist-test-slave-dhph krb5kdc[410](info): commencing operation
krb5kdc: starting...
W20260806 00:58:30.546697 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 00:58:30 dist-test-slave-dhph krb5kdc[410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T00:58:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T00:58:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 00:58:30.703815 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:33817
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:46099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:33817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:30.791270 424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:30.791494 424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:30.791518 424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:30.792984 424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 00:58:30.793043 424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:30.793064 424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:30.793079 424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 00:58:30.793131 424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 00:58:30.794935 424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:46099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:33817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:33817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:30.795228 424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:30.795521 424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:30.801239 432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:30.801476 429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:30.801514 430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:30.801602 424 server_base.cc:1034] running on GCE node
I20260806 00:58:30.801868 424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:30.802204 424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:30.803417 424 hybrid_clock.cc:648] HybridClock initialized: now 1785977910803391 us; error 39 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T00:58:30Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1391 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 00:58:30 dist-test-slave-dhph krb5kdc[437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 00:58:30 dist-test-slave-dhph krb5kdc[437](info): set up 2 sockets
Aug 06 00:58:30 dist-test-slave-dhph krb5kdc[437](info): commencing operation
krb5kdc: starting...
W20260806 00:58:31.938630 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.003s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 00:58:32 dist-test-slave-dhph krb5kdc[437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T00:58:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T00:58:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 00:58:32.083647 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:45237
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:44535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:45237
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:32.171628 450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:32.171854 450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:32.171878 450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:32.173110 450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 00:58:32.173168 450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:32.173183 450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:32.173197 450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 00:58:32.173209 450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 00:58:32.174847 450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:44535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:45237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:45237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:32.175127 450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:32.175494 450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:32.180969 458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:32.180984 456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:32.181020 450 server_base.cc:1034] running on GCE node
W20260806 00:58:32.180984 455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:32.181481 450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:32.181826 450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:32.183085 450 hybrid_clock.cc:648] HybridClock initialized: now 1785977912183066 us; error 30 us; skew 500 ppm
Aug 06 00:58:32 dist-test-slave-dhph krb5kdc[437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977912, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:32.204180 450 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 00:58:32.204870 450 webserver.cc:460] Webserver started at http://127.23.242.62:33211/ using document root <none> and password file <none>
I20260806 00:58:32.205225 450 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:32.205586 450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:32.205950 450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:32.207551 450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "450d15afd5004fe48d7d2a613d292926"
format_stamp: "Formatted at 2026-08-06 00:58:32 on dist-test-slave-dhph"
server_key: "57cfb78a706efc0e7b10245b123572f4"
server_key_iv: "88fa5a91698e14a91fb3b3f930b07b98"
server_key_version: "encryptionkey@0"
I20260806 00:58:32.208037 450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "450d15afd5004fe48d7d2a613d292926"
format_stamp: "Formatted at 2026-08-06 00:58:32 on dist-test-slave-dhph"
server_key: "57cfb78a706efc0e7b10245b123572f4"
server_key_iv: "88fa5a91698e14a91fb3b3f930b07b98"
server_key_version: "encryptionkey@0"
I20260806 00:58:32.210104 450 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260806 00:58:32.211593 465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:32.211971 450 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:32.212073 450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "450d15afd5004fe48d7d2a613d292926"
format_stamp: "Formatted at 2026-08-06 00:58:32 on dist-test-slave-dhph"
server_key: "57cfb78a706efc0e7b10245b123572f4"
server_key_iv: "88fa5a91698e14a91fb3b3f930b07b98"
server_key_version: "encryptionkey@0"
I20260806 00:58:32.212231 450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:32.229689 450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:32.232342 450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:32.232522 450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:32.237602 450 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:45237
I20260806 00:58:32.237630 519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:45237 every 8 connection(s)
I20260806 00:58:32.238080 450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 00:58:32.239562 520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:32.241017 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 450
I20260806 00:58:32.241158 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 00:58:32.241353 24520 external_mini_cluster.cc:1428] Setting key 7de59da05a44d624513a0e71381f58de
I20260806 00:58:32.243021 520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926: Bootstrap starting.
I20260806 00:58:32.243822 520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:32.244169 520 log.cc:824] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:32.244962 520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926: No bootstrap required, opened a new log
I20260806 00:58:32.247390 520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450d15afd5004fe48d7d2a613d292926" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 45237 } }
I20260806 00:58:32.247596 520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:32.247654 520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 450d15afd5004fe48d7d2a613d292926, State: Initialized, Role: FOLLOWER
I20260806 00:58:32.247867 520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450d15afd5004fe48d7d2a613d292926" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 45237 } }
I20260806 00:58:32.247979 520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 00:58:32.248040 520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 00:58:32.248095 520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:32.248963 520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450d15afd5004fe48d7d2a613d292926" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 45237 } }
I20260806 00:58:32.249109 520 leader_election.cc:302] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 450d15afd5004fe48d7d2a613d292926; no voters:
I20260806 00:58:32.249312 520 leader_election.cc:288] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [CANDIDATE]: Term 1 election: Requested vote from peers
Aug 06 00:58:32 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:32.249429 525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:32.249635 520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 00:58:32.249640 525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [term 1 LEADER]: Becoming Leader. State: Replica: 450d15afd5004fe48d7d2a613d292926, State: Running, Role: LEADER
I20260806 00:58:32.249874 525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450d15afd5004fe48d7d2a613d292926" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 45237 } }
I20260806 00:58:32.250435 526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "450d15afd5004fe48d7d2a613d292926" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450d15afd5004fe48d7d2a613d292926" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 45237 } } }
I20260806 00:58:32.250442 527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 450d15afd5004fe48d7d2a613d292926. Latest consensus state: current_term: 1 leader_uuid: "450d15afd5004fe48d7d2a613d292926" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450d15afd5004fe48d7d2a613d292926" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 45237 } } }
I20260806 00:58:32.250671 526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:32.250689 527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:32.251382 537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 00:58:32.252806 537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 00:58:32.254036 523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:32.242007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50266 (local address 127.23.242.62:45237)
0806 00:58:32.242200 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:32.242208 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:32.242235 (+ 27us) server_negotiation.cc:407] Connection header received
0806 00:58:32.242813 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:32.242822 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:32.242874 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:32.243015 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:32.243923 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:32.245032 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:32.245935 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:32.246422 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:32.246563 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:32.250031 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:32.250035 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:32.250036 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:32.250049 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:32.252308 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:32.252759 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:32.252763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:32.252763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:32.252886 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:32.253114 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:32.253115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:32.253115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:32.253367 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:32.253531 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:32.253665 (+ 134us) server_negotiation.cc:299] Negotiation successful
0806 00:58:32.253809 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":101,"threads_started":1}
I20260806 00:58:32.254822 537 catalog_manager.cc:1347] Generated new cluster ID: 91af913f3a5a4555bcc4bdf4720106d1
I20260806 00:58:32.254876 537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 00:58:32.264791 537 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 00:58:32.265380 537 catalog_manager.cc:1504] Loading token signing keys...
I20260806 00:58:32.274776 537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926: Generated new TSK 0
I20260806 00:58:32.275470 537 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.242.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
I20260806 00:58:32.319068 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:0
--local_ip_for_outbound_sockets=127.23.242.1
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:45237
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:44535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:32.441834 547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:32.442071 547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:32.442101 547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:32.443812 547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:32.443867 547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:32.443897 547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 00:58:32.445499 547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:44535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:45237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:32.445791 547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:32.446095 547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:32.452246 555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:32.452170 553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:32.452289 552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:32.452579 547 server_base.cc:1034] running on GCE node
I20260806 00:58:32.452821 547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:32.453054 547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:32.454250 547 hybrid_clock.cc:648] HybridClock initialized: now 1785977912454221 us; error 48 us; skew 500 ppm
Aug 06 00:58:32 dist-test-slave-dhph krb5kdc[437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977912, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:32.457624 547 init.cc:377] Logged in from keytab as kudu/127.23.242.1@KRBTEST.COM (short username kudu)
I20260806 00:58:32.458216 547 webserver.cc:460] Webserver started at http://127.23.242.1:46453/ using document root <none> and password file <none>
I20260806 00:58:32.458688 547 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:32.458787 547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:32.459031 547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:32.460623 547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/data/instance:
uuid: "f35446c781a34a3b94f0c93bc2724e31"
format_stamp: "Formatted at 2026-08-06 00:58:32 on dist-test-slave-dhph"
server_key: "68e6a770f0a1ab838a7ab112b9a82522"
server_key_iv: "9382d629d92fe698f6bc1f107a8edf0c"
server_key_version: "encryptionkey@0"
I20260806 00:58:32.461041 547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance:
uuid: "f35446c781a34a3b94f0c93bc2724e31"
format_stamp: "Formatted at 2026-08-06 00:58:32 on dist-test-slave-dhph"
server_key: "68e6a770f0a1ab838a7ab112b9a82522"
server_key_iv: "9382d629d92fe698f6bc1f107a8edf0c"
server_key_version: "encryptionkey@0"
I20260806 00:58:32.463007 547 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260806 00:58:32.464039 562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:32.464506 547 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:32.464735 547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "f35446c781a34a3b94f0c93bc2724e31"
format_stamp: "Formatted at 2026-08-06 00:58:32 on dist-test-slave-dhph"
server_key: "68e6a770f0a1ab838a7ab112b9a82522"
server_key_iv: "9382d629d92fe698f6bc1f107a8edf0c"
server_key_version: "encryptionkey@0"
I20260806 00:58:32.464866 547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:32.483174 547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:32.485644 547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:32.486017 547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:32.486377 547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:32.486793 547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:32.486855 547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:32.486898 547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:32.486924 547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:32.494936 547 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:43819
I20260806 00:58:32.495014 678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:43819 every 8 connection(s)
I20260806 00:58:32.495383 547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 00:58:32.497978 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 547
I20260806 00:58:32.498171 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance
I20260806 00:58:32.498389 24520 external_mini_cluster.cc:1428] Setting key 42cc8d5ada8b81a9a0509b3893820f08
Aug 06 00:58:32 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977912, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:32.509943 523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:32.496280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:56467 (local address 127.23.242.62:45237)
0806 00:58:32.496343 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:32.496346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:32.496595 (+ 249us) server_negotiation.cc:407] Connection header received
0806 00:58:32.497155 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:32.497156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:32.497169 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:32.497222 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:32.499088 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:32.499850 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:32.501461 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:32.502330 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:32.502425 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:32.506043 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:32.506045 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:32.506045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:32.506056 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:32.508073 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:32.508621 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:32.508623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:32.508623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:32.508693 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:32.509073 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:32.509074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:32.509074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:32.509266 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:32.509357 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:32.509716 (+ 359us) server_negotiation.cc:299] Negotiation successful
0806 00:58:32.509809 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 00:58:32.510075 681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:32.496279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.1:56467)
0806 00:58:32.496468 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:32.496505 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:32.497036 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:32.497415 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:32.497424 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:32.498310 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:32.498843 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:32.498880 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:32.500027 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:32.500036 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:32.501063 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:32.501068 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:32.502482 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:32.502483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:32.502687 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:32.504005 (+ 1318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:32.504015 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:32.505925 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:32.508212 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:32.508220 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:32.508222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:32.508529 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:32.508829 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:32.508830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:32.508830 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:32.508962 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:32.509385 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:32.509392 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:32.509549 (+ 157us) client_negotiation.cc:769] Sending connection context
0806 00:58:32.509671 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 00:58:32.509819 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":94,"threads_started":1}
I20260806 00:58:32.510810 679 heartbeater.cc:347] Connected to a master server at 127.23.242.62:45237
I20260806 00:58:32.510933 679 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:32.511161 679 heartbeater.cc:511] Master 127.23.242.62:45237 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.242.2@KRBTEST.COM; defaulting to no policy
I20260806 00:58:32.511869 483 ts_manager.cc:194] Registered new tserver with Master: f35446c781a34a3b94f0c93bc2724e31 (127.23.242.1:43819)
I20260806 00:58:32.512495 483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.1@KRBTEST.COM'} at 127.23.242.1:56467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
I20260806 00:58:32.534922 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:0
--local_ip_for_outbound_sockets=127.23.242.2
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:45237
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:44535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:32.635356 684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:32.635601 684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:32.635628 684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:32.636868 684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:32.636919 684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:32.636947 684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 00:58:32.638434 684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:44535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:45237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:32.638700 684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:32.638957 684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:32.644426 689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:32.644456 690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:32.644426 692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:32.644919 684 server_base.cc:1034] running on GCE node
I20260806 00:58:32.645120 684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:32.645367 684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:32.646536 684 hybrid_clock.cc:648] HybridClock initialized: now 1785977912646521 us; error 27 us; skew 500 ppm
Aug 06 00:58:32 dist-test-slave-dhph krb5kdc[437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977912, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:32.649366 684 init.cc:377] Logged in from keytab as kudu/127.23.242.2@KRBTEST.COM (short username kudu)
I20260806 00:58:32.650152 684 webserver.cc:460] Webserver started at http://127.23.242.2:43163/ using document root <none> and password file <none>
I20260806 00:58:32.650441 684 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:32.650492 684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:32.650681 684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:32.651938 684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/data/instance:
uuid: "30c31655cfcb4d84b87d9ed34f1d89f7"
format_stamp: "Formatted at 2026-08-06 00:58:32 on dist-test-slave-dhph"
server_key: "fc016dad024a9413e70f14ea00f43b8c"
server_key_iv: "46725be898e9784ee5a337657c4b1ea7"
server_key_version: "encryptionkey@0"
I20260806 00:58:32.652415 684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance:
uuid: "30c31655cfcb4d84b87d9ed34f1d89f7"
format_stamp: "Formatted at 2026-08-06 00:58:32 on dist-test-slave-dhph"
server_key: "fc016dad024a9413e70f14ea00f43b8c"
server_key_iv: "46725be898e9784ee5a337657c4b1ea7"
server_key_version: "encryptionkey@0"
I20260806 00:58:32.654026 684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260806 00:58:32.655098 699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:32.655457 684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 00:58:32.655557 684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "30c31655cfcb4d84b87d9ed34f1d89f7"
format_stamp: "Formatted at 2026-08-06 00:58:32 on dist-test-slave-dhph"
server_key: "fc016dad024a9413e70f14ea00f43b8c"
server_key_iv: "46725be898e9784ee5a337657c4b1ea7"
server_key_version: "encryptionkey@0"
I20260806 00:58:32.655676 684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:32.683456 684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:32.686137 684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:32.686287 684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:32.686688 684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:32.687083 684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:32.687152 684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:32.687207 684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:32.687242 684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:32.694494 684 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:36655
I20260806 00:58:32.694545 816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:36655 every 8 connection(s)
I20260806 00:58:32.694962 684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
Aug 06 00:58:32 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977912, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:32.702286 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 684
I20260806 00:58:32.702409 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance
I20260806 00:58:32.702742 24520 external_mini_cluster.cc:1428] Setting key d62b47872860be39cd253ec02ade11a6
I20260806 00:58:32.705997 523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:32.695974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:48215 (local address 127.23.242.62:45237)
0806 00:58:32.696023 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:32.696029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:32.696307 (+ 278us) server_negotiation.cc:407] Connection header received
0806 00:58:32.696904 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:32.696906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:32.696919 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:32.696977 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:32.697960 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:32.698465 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:32.699523 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:32.699721 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:32.699816 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:32.702265 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:32.702266 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:32.702267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:32.702273 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:32.704367 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:32.704833 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:32.704834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:32.704835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:32.704906 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:32.705201 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:32.705202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:32.705202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:32.705370 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:32.705449 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:32.705817 (+ 368us) server_negotiation.cc:299] Negotiation successful
0806 00:58:32.705906 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 00:58:32.706353 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:32.695954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.2:48215)
0806 00:58:32.696151 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:32.696204 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:32.696782 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:32.697046 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:32.697050 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:32.697660 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:32.697856 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:32.697860 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:32.698567 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:32.698568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:32.699402 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:32.699405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:32.699841 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:32.699842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:32.699965 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:32.700538 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:32.700547 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:32.702075 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:32.704470 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:32.704477 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:32.704479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:32.704746 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:32.704998 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:32.704999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:32.705000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:32.705122 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:32.705495 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:32.705502 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:32.705650 (+ 148us) client_negotiation.cc:769] Sending connection context
0806 00:58:32.705778 (+ 128us) client_negotiation.cc:240] Negotiation successful
0806 00:58:32.705953 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
I20260806 00:58:32.707013 817 heartbeater.cc:347] Connected to a master server at 127.23.242.62:45237
I20260806 00:58:32.707134 817 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:32.707326 817 heartbeater.cc:511] Master 127.23.242.62:45237 requested a full tablet report, sending...
I20260806 00:58:32.707753 483 ts_manager.cc:194] Registered new tserver with Master: 30c31655cfcb4d84b87d9ed34f1d89f7 (127.23.242.2:36655)
I20260806 00:58:32.708374 483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.2@KRBTEST.COM'} at 127.23.242.2:48215
WARNING: no policy specified for kudu/127.23.242.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
I20260806 00:58:32.735522 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:0
--local_ip_for_outbound_sockets=127.23.242.3
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:45237
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:44535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:32.823704 822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:32.823944 822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:32.823973 822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:32.825311 822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:32.825372 822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:32.825407 822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 00:58:32.827308 822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:44535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:45237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:32.827644 822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:32.827935 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:32.833902 828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:32.833971 827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:32.833913 831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:32.834389 822 server_base.cc:1034] running on GCE node
I20260806 00:58:32.834679 822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:32.834924 822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:32.836102 822 hybrid_clock.cc:648] HybridClock initialized: now 1785977912836073 us; error 43 us; skew 500 ppm
Aug 06 00:58:32 dist-test-slave-dhph krb5kdc[437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977912, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:32.839367 822 init.cc:377] Logged in from keytab as kudu/127.23.242.3@KRBTEST.COM (short username kudu)
I20260806 00:58:32.839900 822 webserver.cc:460] Webserver started at http://127.23.242.3:37789/ using document root <none> and password file <none>
I20260806 00:58:32.840161 822 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:32.840207 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:32.840378 822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:32.841576 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/data/instance:
uuid: "20c65ba7e88d46c89780cfee3ae2261d"
format_stamp: "Formatted at 2026-08-06 00:58:32 on dist-test-slave-dhph"
server_key: "036f7aa0eac325e1a4f660c5948fa252"
server_key_iv: "902502cc0eda380392e2f2f92ad8a868"
server_key_version: "encryptionkey@0"
I20260806 00:58:32.841948 822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance:
uuid: "20c65ba7e88d46c89780cfee3ae2261d"
format_stamp: "Formatted at 2026-08-06 00:58:32 on dist-test-slave-dhph"
server_key: "036f7aa0eac325e1a4f660c5948fa252"
server_key_iv: "902502cc0eda380392e2f2f92ad8a868"
server_key_version: "encryptionkey@0"
I20260806 00:58:32.843371 822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 00:58:32.844157 838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:32.844509 822 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:32.844657 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "20c65ba7e88d46c89780cfee3ae2261d"
format_stamp: "Formatted at 2026-08-06 00:58:32 on dist-test-slave-dhph"
server_key: "036f7aa0eac325e1a4f660c5948fa252"
server_key_iv: "902502cc0eda380392e2f2f92ad8a868"
server_key_version: "encryptionkey@0"
I20260806 00:58:32.844738 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:32.856916 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:32.859620 822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:32.859817 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:32.860126 822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:32.860492 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:32.860561 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:32.860610 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:32.860646 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:32.868136 822 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:36217
I20260806 00:58:32.868213 957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:36217 every 8 connection(s)
I20260806 00:58:32.868661 822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
I20260806 00:58:32.871856 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 822
I20260806 00:58:32.871989 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance
I20260806 00:58:32.872180 24520 external_mini_cluster.cc:1428] Setting key 2945508ac0e90fcb8edc4aefbea58878
Aug 06 00:58:32 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977912, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:32.879834 523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:32.869493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:52075 (local address 127.23.242.62:45237)
0806 00:58:32.869555 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:32.869558 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:32.869848 (+ 290us) server_negotiation.cc:407] Connection header received
0806 00:58:32.870465 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:32.870466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:32.870479 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:32.870547 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:32.871673 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:32.872415 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:32.873839 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:32.874033 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:32.874111 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:32.876499 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:32.876500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:32.876500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:32.876508 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:32.877980 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:32.878441 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:32.878443 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:32.878443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:32.878551 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:32.879126 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:32.879127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:32.879128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:32.879264 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:32.879337 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:32.879673 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 00:58:32.879762 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 00:58:32.880052 960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:32.869435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.3:52075)
0806 00:58:32.869693 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:32.869747 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:32.870314 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:32.870743 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:32.870748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:32.871357 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:32.871558 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:32.871562 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:32.872548 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:32.872550 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:32.873742 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:32.873744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:32.874162 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:32.874163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:32.874278 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:32.874875 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:32.874883 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:32.876392 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:32.878112 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:32.878118 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:32.878120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:32.878350 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:32.878875 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:32.878876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:32.878877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:32.879017 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:32.879342 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:32.879348 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:32.879485 (+ 137us) client_negotiation.cc:769] Sending connection context
0806 00:58:32.879585 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 00:58:32.879777 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":98,"threads_started":1}
I20260806 00:58:32.880662 958 heartbeater.cc:347] Connected to a master server at 127.23.242.62:45237
I20260806 00:58:32.880795 958 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:32.881008 958 heartbeater.cc:511] Master 127.23.242.62:45237 requested a full tablet report, sending...
I20260806 00:58:32.881531 483 ts_manager.cc:194] Registered new tserver with Master: 20c65ba7e88d46c89780cfee3ae2261d (127.23.242.3:36217)
I20260806 00:58:32.881944 483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.3@KRBTEST.COM'} at 127.23.242.3:52075
I20260806 00:58:32.884963 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.242.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.242.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.242.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.1.keytab.
Entry for principal oryx/127.23.242.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.1.keytab.
WARNING: no policy specified for oryx/127.23.242.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.242.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.242.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.2.keytab.
Entry for principal oryx/127.23.242.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.2.keytab.
WARNING: no policy specified for oryx/127.23.242.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.242.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.242.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.3.keytab.
Entry for principal oryx/127.23.242.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.3.keytab.
I20260806 00:58:32.979244 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 547
I20260806 00:58:32.985006 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 684
I20260806 00:58:32.991153 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 822
I20260806 00:58:32.996995 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 450
I20260806 00:58:33.004055 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:45237
--webserver_interface=127.23.242.62
--webserver_port=33211
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:44535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:45237
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:33.097397 967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:33.097666 967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:33.097707 967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:33.099035 967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 00:58:33.099116 967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:33.099143 967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:33.099167 967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 00:58:33.099195 967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 00:58:33.100812 967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:44535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:45237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:45237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=33211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:33.101122 967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:33.101456 967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:33.106880 976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:33.106854 974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:33.106850 972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:33.107620 967 server_base.cc:1034] running on GCE node
I20260806 00:58:33.107985 967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:33.108618 967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:33.109859 967 hybrid_clock.cc:648] HybridClock initialized: now 1785977913109842 us; error 93 us; skew 500 ppm
Aug 06 00:58:33 dist-test-slave-dhph krb5kdc[437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977913, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:33.113085 967 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 00:58:33.113564 967 webserver.cc:460] Webserver started at http://127.23.242.62:33211/ using document root <none> and password file <none>
I20260806 00:58:33.113875 967 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:33.113966 967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:33.115823 967 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:33.116619 983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:33.116835 967 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:33.116951 967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "450d15afd5004fe48d7d2a613d292926"
format_stamp: "Formatted at 2026-08-06 00:58:32 on dist-test-slave-dhph"
server_key: "57cfb78a706efc0e7b10245b123572f4"
server_key_iv: "88fa5a91698e14a91fb3b3f930b07b98"
server_key_version: "encryptionkey@0"
I20260806 00:58:33.117318 967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:33.144243 967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:33.147056 967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:33.147270 967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:33.152341 967 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:45237
I20260806 00:58:33.152383 1037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:45237 every 8 connection(s)
I20260806 00:58:33.152901 967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 00:58:33.154513 1038 sys_catalog.cc:263] Verifying existing consensus state
I20260806 00:58:33.155160 1038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926: Bootstrap starting.
I20260806 00:58:33.161561 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 967
I20260806 00:58:33.161960 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:43819
--local_ip_for_outbound_sockets=127.23.242.1
--tserver_master_addrs=127.23.242.62:45237
--webserver_port=46453
--webserver_interface=127.23.242.1
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:44535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.242.1
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 00:58:33.165194 1038 log.cc:824] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:33.168045 1038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 00:58:33.168479 1038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926: Bootstrap complete.
I20260806 00:58:33.173864 1038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450d15afd5004fe48d7d2a613d292926" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 45237 } }
I20260806 00:58:33.174504 1038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 450d15afd5004fe48d7d2a613d292926, State: Initialized, Role: FOLLOWER
I20260806 00:58:33.174798 1038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450d15afd5004fe48d7d2a613d292926" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 45237 } }
I20260806 00:58:33.174907 1038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 00:58:33.174983 1038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 00:58:33.175043 1038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [term 1 FOLLOWER]: Advancing to term 2
I20260806 00:58:33.177287 1038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450d15afd5004fe48d7d2a613d292926" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 45237 } }
I20260806 00:58:33.177454 1038 leader_election.cc:302] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 450d15afd5004fe48d7d2a613d292926; no voters:
I20260806 00:58:33.177641 1038 leader_election.cc:288] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 00:58:33.177825 1042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 00:58:33.177927 1038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 00:58:33.178045 1042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [term 2 LEADER]: Becoming Leader. State: Replica: 450d15afd5004fe48d7d2a613d292926, State: Running, Role: LEADER
I20260806 00:58:33.178191 1042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450d15afd5004fe48d7d2a613d292926" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 45237 } }
I20260806 00:58:33.178555 1043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "450d15afd5004fe48d7d2a613d292926" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450d15afd5004fe48d7d2a613d292926" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 45237 } } }
I20260806 00:58:33.178666 1043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:33.178833 1044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 450d15afd5004fe48d7d2a613d292926. Latest consensus state: current_term: 2 leader_uuid: "450d15afd5004fe48d7d2a613d292926" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "450d15afd5004fe48d7d2a613d292926" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 45237 } } }
I20260806 00:58:33.178898 1044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:33.179260 1051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 00:58:33.180007 1051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 00:58:33.180157 1051 catalog_manager.cc:1259] Loaded cluster ID: 91af913f3a5a4555bcc4bdf4720106d1
I20260806 00:58:33.180198 1051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 00:58:33.181344 1051 catalog_manager.cc:1504] Loading token signing keys...
I20260806 00:58:33.181538 1051 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 450d15afd5004fe48d7d2a613d292926: Loaded TSK: 0
I20260806 00:58:33.181761 1051 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 00:58:33.260317 1040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:33.260583 1040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:33.260625 1040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:33.261824 1040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:33.261899 1040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:33.261947 1040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 00:58:33.263501 1040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:44535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.1.keytab
--principal=oryx/127.23.242.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:43819
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=46453
--tserver_master_addrs=127.23.242.62:45237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:33.263785 1040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:33.264034 1040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:33.271054 1067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:33.271047 1065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:33.271044 1064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:33.271279 1040 server_base.cc:1034] running on GCE node
I20260806 00:58:33.271478 1040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:33.271819 1040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:33.273010 1040 hybrid_clock.cc:648] HybridClock initialized: now 1785977913272985 us; error 41 us; skew 500 ppm
Aug 06 00:58:33 dist-test-slave-dhph krb5kdc[437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977913, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:33.276211 1040 init.cc:377] Logged in from keytab as oryx/127.23.242.1@KRBTEST.COM (short username oryx)
I20260806 00:58:33.276731 1040 webserver.cc:460] Webserver started at http://127.23.242.1:46453/ using document root <none> and password file <none>
I20260806 00:58:33.276971 1040 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:33.277060 1040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:33.278776 1040 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 00:58:33.279780 1074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:33.280247 1040 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:33.280364 1040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "f35446c781a34a3b94f0c93bc2724e31"
format_stamp: "Formatted at 2026-08-06 00:58:32 on dist-test-slave-dhph"
server_key: "68e6a770f0a1ab838a7ab112b9a82522"
server_key_iv: "9382d629d92fe698f6bc1f107a8edf0c"
server_key_version: "encryptionkey@0"
I20260806 00:58:33.280761 1040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:33.314940 1040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:33.317780 1040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:33.317972 1040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:33.318241 1040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:33.318682 1040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:33.318740 1040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:33.318785 1040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:33.318811 1040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:33.327011 1040 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:43819
I20260806 00:58:33.327055 1191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:43819 every 8 connection(s)
I20260806 00:58:33.327600 1040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 00:58:33.329825 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 1040
I20260806 00:58:33.330155 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:36655
--local_ip_for_outbound_sockets=127.23.242.2
--tserver_master_addrs=127.23.242.62:45237
--webserver_port=43163
--webserver_interface=127.23.242.2
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:44535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.242.2
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 00:58:33 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:33 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:33.337764 1194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:33.328513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.1:47587)
0806 00:58:33.328709 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:33.328762 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:33.329254 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:33.329873 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:33.329878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:33.330655 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:33.330862 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:33.330867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.331993 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.331995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:33.332822 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:33.332826 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.333439 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.333440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:33.333585 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:33.334371 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:33.334381 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:33.337494 (+ 3113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":108,"threads_started":1}
W20260806 00:58:33.337814 1195 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:33.328620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:47587 (local address 127.23.242.62:45237)
0806 00:58:33.328833 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:33.328842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:33.328880 (+ 38us) server_negotiation.cc:407] Connection header received
0806 00:58:33.329493 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:33.329498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:33.329543 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:33.329834 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:33.330995 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.331878 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.332924 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.333341 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.333462 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:33.337659 (+ 4197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.1:47587: BlockingRecv error: recv got EOF from 127.23.242.1:47587 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":130,"threads_started":1}
W20260806 00:58:33.337990 1192 heartbeater.cc:644] Failed to heartbeat to 127.23.242.62:45237 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.242.62:45237: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Aug 06 00:58:33 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:33 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:33.345849 1194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:33.338569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.1:57981)
0806 00:58:33.338786 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:33.338801 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:33.338886 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:33.339761 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:33.339762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:33.339990 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:33.340239 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:33.340248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.341068 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.341069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:33.342150 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:33.342157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.342610 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.342611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:33.342694 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:33.343180 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:33.343187 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:33.345762 (+ 2575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190}
W20260806 00:58:33.346272 1195 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:33.338764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:57981 (local address 127.23.242.62:45237)
0806 00:58:33.338808 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:33.338810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:33.338897 (+ 87us) server_negotiation.cc:407] Connection header received
0806 00:58:33.339024 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:33.339025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:33.339046 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:33.339102 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:33.340412 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.340957 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.342282 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.342493 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.342563 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:33.346155 (+ 3592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.1:57981: BlockingRecv error: recv got EOF from 127.23.242.1:57981 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Aug 06 00:58:33 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:33 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:33.352088 1194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:33.346476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.1:56035)
0806 00:58:33.346544 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:33.346558 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:33.346658 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:33.346873 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:33.346875 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:33.347003 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:33.347045 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:33.347046 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.348038 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.348039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:33.348664 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:33.348667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.349153 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.349154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:33.349200 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:33.349722 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:33.349727 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:33.352031 (+ 2304us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 00:58:33.352236 1195 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:33.346554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:56035 (local address 127.23.242.62:45237)
0806 00:58:33.346662 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:33.346664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:33.346679 (+ 15us) server_negotiation.cc:407] Connection header received
0806 00:58:33.346712 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:33.346713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:33.346730 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:33.346809 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:33.347137 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.347939 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.348747 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.349052 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.349122 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:33.352160 (+ 3038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.1:56035: BlockingRecv error: recv got EOF from 127.23.242.1:56035 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260806 00:58:33.352351 1192 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 00:58:33.425908 1196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:33.426133 1196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:33.426160 1196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:33.427465 1196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:33.427518 1196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:33.427548 1196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 00:58:33.429401 1196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:44535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.2.keytab
--principal=oryx/127.23.242.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:36655
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=43163
--tserver_master_addrs=127.23.242.62:45237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:33.429699 1196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:33.429961 1196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:33.435801 1201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:33.435803 1204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:33.435794 1202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:33.435971 1196 server_base.cc:1034] running on GCE node
I20260806 00:58:33.436304 1196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:33.436528 1196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:33.437716 1196 hybrid_clock.cc:648] HybridClock initialized: now 1785977913437730 us; error 63 us; skew 500 ppm
Aug 06 00:58:33 dist-test-slave-dhph krb5kdc[437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977913, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:33.440742 1196 init.cc:377] Logged in from keytab as oryx/127.23.242.2@KRBTEST.COM (short username oryx)
I20260806 00:58:33.441272 1196 webserver.cc:460] Webserver started at http://127.23.242.2:43163/ using document root <none> and password file <none>
I20260806 00:58:33.441512 1196 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:33.441596 1196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:33.443040 1196 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 00:58:33.443928 1211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:33.444164 1196 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:33.444257 1196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "30c31655cfcb4d84b87d9ed34f1d89f7"
format_stamp: "Formatted at 2026-08-06 00:58:32 on dist-test-slave-dhph"
server_key: "fc016dad024a9413e70f14ea00f43b8c"
server_key_iv: "46725be898e9784ee5a337657c4b1ea7"
server_key_version: "encryptionkey@0"
I20260806 00:58:33.444669 1196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:33.460148 1196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:33.462760 1196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:33.462947 1196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:33.463239 1196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:33.463821 1196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:33.463898 1196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:33.463951 1196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:33.463999 1196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:33.471140 1196 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:36655
I20260806 00:58:33.471231 1326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:36655 every 8 connection(s)
I20260806 00:58:33.471596 1196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 00:58:33.476567 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 1196
I20260806 00:58:33.476933 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:36217
--local_ip_for_outbound_sockets=127.23.242.3
--tserver_master_addrs=127.23.242.62:45237
--webserver_port=37789
--webserver_interface=127.23.242.3
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:44535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.242.3
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 00:58:33 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:33 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 00:58:33.480654 1195 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:33.472473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:54093 (local address 127.23.242.62:45237)
0806 00:58:33.472556 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:33.472558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:33.472779 (+ 221us) server_negotiation.cc:407] Connection header received
0806 00:58:33.473234 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:33.473235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:33.473250 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:33.473314 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:33.474328 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.474895 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.476096 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.476313 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.476377 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:33.480598 (+ 4221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.2:54093: BlockingRecv error: recv got EOF from 127.23.242.2:54093 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 00:58:33.480749 1329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:33.472448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.2:54093)
0806 00:58:33.472613 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:33.472664 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:33.473140 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:33.473365 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:33.473368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:33.474006 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:33.474210 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:33.474215 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.475005 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.475006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:33.475995 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:33.475998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.476417 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.476418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:33.476578 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:33.477287 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:33.477297 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:33.480461 (+ 3164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
W20260806 00:58:33.481009 1327 heartbeater.cc:644] Failed to heartbeat to 127.23.242.62:45237 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.242.62:45237: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Aug 06 00:58:33 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:33 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:33.486203 1329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:33.481275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.2:58565)
0806 00:58:33.481357 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:33.481370 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:33.481424 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:33.481562 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:33.481562 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:33.481694 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:33.481754 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:33.481755 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.482793 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.482794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:33.483343 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:33.483345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.483907 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.483908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:33.483967 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:33.484438 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:33.484443 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:33.486153 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 00:58:33.486373 1195 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:33.481361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:58565 (local address 127.23.242.62:45237)
0806 00:58:33.481425 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:33.481428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:33.481438 (+ 10us) server_negotiation.cc:407] Connection header received
0806 00:58:33.481475 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:33.481475 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:33.481488 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:33.481549 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:33.481833 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.482700 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.483430 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.483829 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.483910 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:33.486285 (+ 2375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.2:58565: BlockingRecv error: recv got EOF from 127.23.242.2:58565 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 06 00:58:33 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:33 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:33.491326 1329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:33.486702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.2:46291)
0806 00:58:33.486754 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:33.486764 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:33.486863 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:33.487048 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:33.487049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:33.487193 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:33.487221 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:33.487223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.487884 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.487885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:33.488376 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:33.488378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.488733 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.488734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:33.488788 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:33.489342 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:33.489347 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:33.491235 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 00:58:33.491536 1195 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:33.486771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:46291 (local address 127.23.242.62:45237)
0806 00:58:33.486870 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:33.486872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:33.486884 (+ 12us) server_negotiation.cc:407] Connection header received
0806 00:58:33.486954 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:33.486955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:33.486966 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:33.487029 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:33.487299 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.487803 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.488456 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.488649 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.488717 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:33.491320 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.2:46291: BlockingRecv error: recv got EOF from 127.23.242.2:46291 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260806 00:58:33.491546 1327 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 00:58:33.568756 1330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:33.569033 1330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:33.569077 1330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:33.570276 1330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:33.570353 1330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:33.570402 1330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 00:58:33.571942 1330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:44535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/krb5kdc/oryx_127.23.242.3.keytab
--principal=oryx/127.23.242.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:36217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=37789
--tserver_master_addrs=127.23.242.62:45237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:33.572229 1330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:33.572472 1330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:33.577764 1335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:33.577741 1338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:33.577730 1336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:33.577966 1330 server_base.cc:1034] running on GCE node
I20260806 00:58:33.578311 1330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:33.578668 1330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:33.579849 1330 hybrid_clock.cc:648] HybridClock initialized: now 1785977913579793 us; error 73 us; skew 500 ppm
Aug 06 00:58:33 dist-test-slave-dhph krb5kdc[437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977913, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.242.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:33.582716 1330 init.cc:377] Logged in from keytab as oryx/127.23.242.3@KRBTEST.COM (short username oryx)
I20260806 00:58:33.583221 1330 webserver.cc:460] Webserver started at http://127.23.242.3:37789/ using document root <none> and password file <none>
I20260806 00:58:33.583434 1330 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:33.583492 1330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:33.585047 1330 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:33.585691 1345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:33.585892 1330 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 00:58:33.585975 1330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "20c65ba7e88d46c89780cfee3ae2261d"
format_stamp: "Formatted at 2026-08-06 00:58:32 on dist-test-slave-dhph"
server_key: "036f7aa0eac325e1a4f660c5948fa252"
server_key_iv: "902502cc0eda380392e2f2f92ad8a868"
server_key_version: "encryptionkey@0"
I20260806 00:58:33.586345 1330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:33.595587 1330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:33.597743 1330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:33.597869 1330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:33.598142 1330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:33.598497 1330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:33.598541 1330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:33.598574 1330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:33.598639 1330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:33.604377 1330 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:36217
I20260806 00:58:33.604494 1459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:36217 every 8 connection(s)
I20260806 00:58:33.605170 1330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
Aug 06 00:58:33 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:33 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 00:58:33.614144 1195 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:33.606040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:52473 (local address 127.23.242.62:45237)
0806 00:58:33.606104 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:33.606106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:33.606355 (+ 249us) server_negotiation.cc:407] Connection header received
0806 00:58:33.607414 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:33.607416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:33.607427 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:33.607492 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:33.608630 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.609166 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.610295 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.610494 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.610563 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:33.614093 (+ 3530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.3:52473: BlockingRecv error: recv got EOF from 127.23.242.3:52473 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 00:58:33.614183 1462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:33.605968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.3:52473)
0806 00:58:33.606201 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:33.606239 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:33.607289 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:33.607747 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:33.607755 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:33.608190 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:33.608459 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:33.608467 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.609311 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.609313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:33.610178 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:33.610181 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.610641 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.610642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:33.610811 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:33.611385 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:33.611393 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:33.613988 (+ 2595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":72,"threads_started":1}
W20260806 00:58:33.614562 1460 heartbeater.cc:644] Failed to heartbeat to 127.23.242.62:45237 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.242.62:45237: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
I20260806 00:58:33.614676 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 1330
Aug 06 00:58:33 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:33 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:33.619608 1462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:33.614922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.3:56159)
0806 00:58:33.614975 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:33.614988 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:33.615041 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:33.615285 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:33.615287 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:33.615421 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:33.615473 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:33.615474 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.616115 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.616116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:33.616618 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:33.616619 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.616989 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.616990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:33.617062 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:33.617571 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:33.617585 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:33.619534 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260806 00:58:33.619724 1195 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:33.615127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:56159 (local address 127.23.242.62:45237)
0806 00:58:33.615170 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:33.615172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:33.615180 (+ 8us) server_negotiation.cc:407] Connection header received
0806 00:58:33.615205 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:33.615206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:33.615217 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:33.615268 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:33.615549 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.616034 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.616701 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.616903 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.616987 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:33.619655 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.3:56159: BlockingRecv error: recv got EOF from 127.23.242.3:56159 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 00:58:33.623297 1464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:33.615141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50270 (local address 127.23.242.62:45237)
0806 00:58:33.615263 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:33.615265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:33.615291 (+ 26us) server_negotiation.cc:407] Connection header received
0806 00:58:33.615393 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:33.615393 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:33.615405 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:33.615469 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:33.616243 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.616801 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.617731 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.617928 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.617998 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:33.619398 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:33.619402 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:33.619403 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:33.619414 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:33.621675 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:33.622082 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:33.622086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:33.622086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:33.622187 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:33.622410 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:33.622411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:33.622412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:33.622756 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:33.622914 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:33.623073 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 00:58:33.623170 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
Aug 06 00:58:33 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:33 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:33.625058 1462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:33.619874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.3:38895)
0806 00:58:33.620050 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:33.620066 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:33.620135 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:33.620312 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:33.620314 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:33.620424 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:33.620453 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:33.620454 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.621116 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.621117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:33.621902 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:33.621904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.622391 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.622393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:33.622459 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:33.622901 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:33.622907 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:33.625008 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161}
W20260806 00:58:33.625201 1460 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 00:58:33.625344 1195 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:33.619927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:38895 (local address 127.23.242.62:45237)
0806 00:58:33.619989 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:33.619991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:33.620136 (+ 145us) server_negotiation.cc:407] Connection header received
0806 00:58:33.620181 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:33.620182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:33.620193 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:33.620268 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:33.620516 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.621035 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.621980 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:33.622266 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:33.622388 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:33.625153 (+ 2765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.3:38895: BlockingRecv error: recv got EOF from 127.23.242.3:38895 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 06 00:58:34 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:34 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:34.358054 1465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:34.352820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.1:49035)
0806 00:58:34.352970 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:34.352981 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:34.353030 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:34.353198 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:34.353199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:34.353367 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:34.353419 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:34.353420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:34.354106 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:34.354108 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:34.354761 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:34.354763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:34.355179 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:34.355180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:34.355240 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:34.355809 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:34.355813 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:34.357982 (+ 2169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
W20260806 00:58:34.358136 1466 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:34.352924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:49035 (local address 127.23.242.62:45237)
0806 00:58:34.353069 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:34.353070 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:34.353078 (+ 8us) server_negotiation.cc:407] Connection header received
0806 00:58:34.353098 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:34.353099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:34.353114 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:34.353185 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:34.353496 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:34.354023 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:34.354866 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:34.355064 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:34.355130 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:34.358076 (+ 2946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.1:49035: BlockingRecv error: recv got EOF from 127.23.242.1:49035 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
Aug 06 00:58:34 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:34 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:34.497745 1467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:34.491986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.2:38711)
0806 00:58:34.492180 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:34.492192 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:34.492280 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:34.492462 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:34.492463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:34.492673 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:34.492725 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:34.492727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:34.493473 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:34.493474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:34.494064 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:34.494066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:34.494480 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:34.494481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:34.494550 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:34.495196 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:34.495201 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:34.497660 (+ 2459us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":92,"threads_started":1}
W20260806 00:58:34.497867 1466 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:34.492152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:38711 (local address 127.23.242.62:45237)
0806 00:58:34.492254 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:34.492256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:34.492269 (+ 13us) server_negotiation.cc:407] Connection header received
0806 00:58:34.492355 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:34.492356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:34.492370 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:34.492454 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:34.492820 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:34.493379 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:34.494175 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:34.494387 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:34.494476 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:34.497785 (+ 3309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.2:38711: BlockingRecv error: recv got EOF from 127.23.242.2:38711 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 06 00:58:34 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:34 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:34.631470 1468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:34.625645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.3:55463)
0806 00:58:34.625857 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:34.625870 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:34.625964 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:34.626130 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:34.626130 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:34.626353 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:34.626405 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:34.626407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:34.627380 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:34.627382 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:34.627943 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:34.627945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:34.628452 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:34.628453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:34.628521 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:34.629031 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:34.629041 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:34.631387 (+ 2346us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":120,"threads_started":1}
W20260806 00:58:34.631624 1466 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:34.625759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:55463 (local address 127.23.242.62:45237)
0806 00:58:34.625851 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:34.625854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:34.625943 (+ 89us) server_negotiation.cc:407] Connection header received
0806 00:58:34.626037 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:34.626038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:34.626053 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:34.626122 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:34.626489 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:34.627271 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:34.628045 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:34.628334 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:34.628437 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:34.631523 (+ 3086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.3:55463: BlockingRecv error: recv got EOF from 127.23.242.3:55463 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 00:58:35 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:35 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:35.366504 1469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:35.358866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.1:44559)
0806 00:58:35.359040 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:35.359051 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:35.359146 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:35.359408 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:35.359409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:35.359575 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:35.359640 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:35.359641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:35.360728 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:35.360730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:35.361580 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:35.361585 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:35.363077 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:35.363080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:35.363201 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:35.363937 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:35.363943 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:35.366434 (+ 2491us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":112,"threads_started":1}
W20260806 00:58:35.366780 1470 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:35.358997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:44559 (local address 127.23.242.62:45237)
0806 00:58:35.359123 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:35.359125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:35.359135 (+ 10us) server_negotiation.cc:407] Connection header received
0806 00:58:35.359287 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:35.359288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:35.359302 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:35.359377 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:35.359838 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:35.360578 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:35.361988 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:35.362537 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:35.362740 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:35.366689 (+ 3949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.1:44559: BlockingRecv error: recv got EOF from 127.23.242.1:44559 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
Aug 06 00:58:35 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:35 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:35.504487 1471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:35.498314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.2:54249)
0806 00:58:35.498441 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:35.498454 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:35.498516 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:35.498703 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:35.498704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:35.499173 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:35.499232 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:35.499235 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:35.500386 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:35.500387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:35.501095 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:35.501098 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:35.501540 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:35.501542 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:35.501592 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:35.502260 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:35.502265 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:35.504419 (+ 2154us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":87,"threads_started":1}
W20260806 00:58:35.504668 1470 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:35.498456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:54249 (local address 127.23.242.62:45237)
0806 00:58:35.498530 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:35.498532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:35.498543 (+ 11us) server_negotiation.cc:407] Connection header received
0806 00:58:35.498569 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:35.498570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:35.498600 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:35.498667 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:35.499344 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:35.500039 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:35.501222 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:35.501443 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:35.501520 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:35.504564 (+ 3044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.2:54249: BlockingRecv error: recv got EOF from 127.23.242.2:54249 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 06 00:58:35 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:35 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:35.638062 1472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:35.632294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.3:49543)
0806 00:58:35.632490 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:35.632506 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:35.632580 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:35.632789 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:35.632790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:35.632981 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:35.633034 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:35.633035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:35.634319 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:35.634322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:35.635051 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:35.635055 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:35.635592 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:35.635593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:35.635666 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:35.636191 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:35.636196 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:35.637999 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":90,"threads_started":1}
W20260806 00:58:35.638229 1470 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:35.632479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:49543 (local address 127.23.242.62:45237)
0806 00:58:35.632566 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:35.632568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:35.632582 (+ 14us) server_negotiation.cc:407] Connection header received
0806 00:58:35.632643 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:35.632644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:35.632658 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:35.632719 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:35.633418 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:35.634087 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:35.635218 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:35.635455 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:35.635536 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:35.638140 (+ 2604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.3:49543: BlockingRecv error: recv got EOF from 127.23.242.3:49543 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 00:58:36 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:36 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:36.372884 1473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:36.367309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.1:40163)
0806 00:58:36.367498 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:36.367515 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:36.367594 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:36.367857 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:36.367858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:36.368033 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:36.368087 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:36.368088 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:36.368907 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:36.368908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:36.369536 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:36.369539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:36.370182 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:36.370184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:36.370250 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:36.370809 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:36.370813 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:36.372805 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":75,"threads_started":1}
W20260806 00:58:36.372957 1474 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:36.367381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:40163 (local address 127.23.242.62:45237)
0806 00:58:36.367567 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:36.367569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:36.367580 (+ 11us) server_negotiation.cc:407] Connection header received
0806 00:58:36.367717 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:36.367718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:36.367732 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:36.367811 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:36.368231 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:36.368812 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:36.369718 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:36.370080 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:36.370177 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:36.372900 (+ 2723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.1:40163: BlockingRecv error: recv got EOF from 127.23.242.1:40163 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":113,"threads_started":1}
Aug 06 00:58:36 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:36 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:36.511085 1475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:36.505228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.2:34971)
0806 00:58:36.505434 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:36.505450 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:36.505533 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:36.505948 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:36.505949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:36.506149 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:36.506203 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:36.506205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:36.507033 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:36.507034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:36.507778 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:36.507782 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:36.508329 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:36.508330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:36.508381 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:36.508906 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:36.508911 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:36.511004 (+ 2093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":113,"threads_started":1}
W20260806 00:58:36.511279 1474 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:36.505333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:34971 (local address 127.23.242.62:45237)
0806 00:58:36.505404 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:36.505406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:36.505667 (+ 261us) server_negotiation.cc:407] Connection header received
0806 00:58:36.505742 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:36.505742 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:36.505765 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:36.506065 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:36.506307 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:36.506925 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:36.507910 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:36.508174 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:36.508333 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:36.511211 (+ 2878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.2:34971: BlockingRecv error: recv got EOF from 127.23.242.2:34971 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 06 00:58:36 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:36 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:36.646061 1476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:36.638836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.3:58215)
0806 00:58:36.639085 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:36.639106 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:36.639227 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:36.639504 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:36.639506 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:36.639738 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:36.639808 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:36.639809 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:36.640778 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:36.640780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:36.641525 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:36.641533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:36.642544 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:36.642545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:36.642648 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:36.643444 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:36.643450 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:36.645974 (+ 2524us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":83,"threads_started":1}
W20260806 00:58:36.646180 1474 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:36.639066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:58215 (local address 127.23.242.62:45237)
0806 00:58:36.639204 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:36.639206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:36.639225 (+ 19us) server_negotiation.cc:407] Connection header received
0806 00:58:36.639369 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:36.639370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:36.639388 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:36.639446 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:36.640024 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:36.640642 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:36.641712 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:36.642403 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:36.642512 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:36.646116 (+ 3604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.3:58215: BlockingRecv error: recv got EOF from 127.23.242.3:58215 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Aug 06 00:58:37 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:37 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:37.384634 1477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:37.373459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.1:52251)
0806 00:58:37.373665 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:37.373679 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:37.373759 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:37.373979 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:37.373980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:37.374177 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:37.374243 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:37.374245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:37.375272 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:37.375273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:37.375911 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:37.375913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:37.376311 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:37.376312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:37.376361 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:37.382483 (+ 6122us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:37.382489 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:37.384570 (+ 2081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":111,"threads_started":1}
W20260806 00:58:37.384873 1478 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:37.373564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:52251 (local address 127.23.242.62:45237)
0806 00:58:37.373703 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:37.373705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:37.373738 (+ 33us) server_negotiation.cc:407] Connection header received
0806 00:58:37.373870 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:37.373871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:37.373885 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:37.373952 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:37.374348 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:37.375163 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:37.376009 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:37.376217 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:37.376288 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:37.384780 (+ 8492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.1:52251: BlockingRecv error: recv got EOF from 127.23.242.1:52251 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
Aug 06 00:58:37 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:37 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:37.520227 1479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:37.511676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.2:39567)
0806 00:58:37.511826 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:37.511841 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:37.511920 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:37.512136 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:37.512137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:37.512375 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:37.512425 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:37.512427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:37.513274 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:37.513274 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:37.513833 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:37.513836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:37.514311 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:37.514313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:37.514364 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:37.517844 (+ 3480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:37.517850 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:37.520164 (+ 2314us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
W20260806 00:58:37.520383 1478 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:37.511770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:39567 (local address 127.23.242.62:45237)
0806 00:58:37.511855 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:37.511857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:37.511971 (+ 114us) server_negotiation.cc:407] Connection header received
0806 00:58:37.512010 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:37.512011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:37.512028 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:37.512076 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:37.512507 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:37.513186 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:37.513938 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:37.514163 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:37.514244 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:37.520305 (+ 6061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.2:39567: BlockingRecv error: recv got EOF from 127.23.242.2:39567 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 06 00:58:37 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:37 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:37.652251 1480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:37.646715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.3:41107)
0806 00:58:37.646859 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:37.646873 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:37.646946 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:37.647243 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:37.647244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:37.647475 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:37.647537 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:37.647539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:37.648269 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:37.648270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:37.648832 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:37.648834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:37.649192 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:37.649193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:37.649249 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:37.649864 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:37.649868 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:37.652186 (+ 2318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
W20260806 00:58:37.652417 1478 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:37.647023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:41107 (local address 127.23.242.62:45237)
0806 00:58:37.647090 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:37.647093 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:37.647101 (+ 8us) server_negotiation.cc:407] Connection header received
0806 00:58:37.647126 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:37.647128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:37.647143 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:37.647203 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:37.647689 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:37.648174 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:37.648918 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:37.649115 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:37.649172 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:37.652344 (+ 3172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.3:41107: BlockingRecv error: recv got EOF from 127.23.242.3:41107 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Aug 06 00:58:38 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:38 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:38.390739 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:38.385190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.1:41803)
0806 00:58:38.385380 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:38.385392 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:38.385473 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:38.385645 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:38.385646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:38.385819 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:38.385870 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:38.385871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:38.386626 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:38.386627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:38.387231 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:38.387234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:38.387645 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:38.387646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:38.387713 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:38.388288 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:38.388293 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:38.390665 (+ 2372us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":101,"threads_started":1}
W20260806 00:58:38.390913 1482 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:38.385296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:41803 (local address 127.23.242.62:45237)
0806 00:58:38.385415 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:38.385417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:38.385458 (+ 41us) server_negotiation.cc:407] Connection header received
0806 00:58:38.385522 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:38.385523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:38.385534 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:38.385598 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:38.385983 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:38.386503 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:38.387353 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:38.387549 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:38.387617 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:38.390822 (+ 3205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.1:41803: BlockingRecv error: recv got EOF from 127.23.242.1:41803 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
Aug 06 00:58:38 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:38 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:38.527395 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:38.520885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.2:36063)
0806 00:58:38.521029 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:38.521041 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:38.521117 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:38.521321 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:38.521323 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:38.521524 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:38.521574 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:38.521576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:38.522463 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:38.522464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:38.523343 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:38.523346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:38.523991 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:38.523992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:38.524069 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:38.524785 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:38.524791 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:38.527307 (+ 2516us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
W20260806 00:58:38.527494 1482 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:38.521071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:36063 (local address 127.23.242.62:45237)
0806 00:58:38.521143 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:38.521146 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:38.521192 (+ 46us) server_negotiation.cc:407] Connection header received
0806 00:58:38.521219 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:38.521220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:38.521230 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:38.521289 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:38.521849 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:38.522377 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:38.523486 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:38.523867 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:38.523990 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:38.527426 (+ 3436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.2:36063: BlockingRecv error: recv got EOF from 127.23.242.2:36063 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 06 00:58:38 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:38 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:38.658362 1484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:38.652880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.3:44865)
0806 00:58:38.653040 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:38.653055 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:38.653147 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:38.653338 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:38.653339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:38.653520 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:38.653567 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:38.653569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:38.654349 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:38.654351 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:38.655093 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:38.655095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:38.655662 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:38.655663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:38.655717 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:38.656272 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:38.656277 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:38.658295 (+ 2018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":102,"threads_started":1}
W20260806 00:58:38.658522 1482 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:38.652998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:44865 (local address 127.23.242.62:45237)
0806 00:58:38.653055 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:38.653058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:38.653127 (+ 69us) server_negotiation.cc:407] Connection header received
0806 00:58:38.653238 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:38.653240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:38.653256 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:38.653331 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:38.653644 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:38.654236 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:38.655243 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:38.655544 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:38.655613 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:38.658419 (+ 2806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.3:44865: BlockingRecv error: recv got EOF from 127.23.242.3:44865 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Aug 06 00:58:39 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:39 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:39.396852 1485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:39.391247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.1:55229)
0806 00:58:39.391414 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:39.391425 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:39.391484 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:39.391713 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:39.391714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:39.391883 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:39.391935 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:39.391936 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:39.392670 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:39.392671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:39.393280 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:39.393282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:39.393851 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:39.393851 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:39.393900 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:39.394394 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:39.394399 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:39.396778 (+ 2379us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":106,"threads_started":1}
W20260806 00:58:39.397012 1486 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:39.391366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:55229 (local address 127.23.242.62:45237)
0806 00:58:39.391532 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:39.391534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:39.391542 (+ 8us) server_negotiation.cc:407] Connection header received
0806 00:58:39.391563 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:39.391564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:39.391574 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:39.391646 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:39.392027 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:39.392543 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:39.393376 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:39.393575 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:39.393672 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:39.396912 (+ 3240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.1:55229: BlockingRecv error: recv got EOF from 127.23.242.1:55229 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":104,"threads_started":1}
Aug 06 00:58:39 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:39 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 00:58:39.534818 1486 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:39.528276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:50155 (local address 127.23.242.62:45237)
0806 00:58:39.528356 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:39.528359 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:39.528462 (+ 103us) server_negotiation.cc:407] Connection header received
0806 00:58:39.528552 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:39.528554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:39.528571 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:39.528668 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:39.529170 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:39.530161 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:39.531161 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:39.531598 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:39.531698 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:39.534759 (+ 3061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.2:50155: BlockingRecv error: recv got EOF from 127.23.242.2:50155 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 00:58:39.535091 1487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:39.528172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.2:50155)
0806 00:58:39.528350 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:39.528364 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:39.528455 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:39.528725 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:39.528726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:39.528976 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:39.529070 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:39.529072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:39.530378 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:39.530380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:39.531040 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:39.531044 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:39.531729 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:39.531730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:39.531803 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:39.532408 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:39.532413 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:39.534879 (+ 2466us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":106,"threads_started":1}
Aug 06 00:58:39 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:39 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:39.665228 1488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:39.659029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.3:40049)
0806 00:58:39.659191 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:39.659203 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:39.659281 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:39.659462 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:39.659463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:39.659672 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:39.659736 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:39.659738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:39.660677 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:39.660679 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:39.661356 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:39.661359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:39.661797 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:39.661798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:39.661852 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:39.662820 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:39.662825 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:39.665159 (+ 2334us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
W20260806 00:58:39.665405 1486 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:39.659186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:40049 (local address 127.23.242.62:45237)
0806 00:58:39.659251 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:39.659254 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:39.659267 (+ 13us) server_negotiation.cc:407] Connection header received
0806 00:58:39.659363 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:39.659364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:39.659385 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:39.659451 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:39.659896 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:39.660551 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:39.661472 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:39.661693 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:39.661766 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:39.665314 (+ 3548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.3:40049: BlockingRecv error: recv got EOF from 127.23.242.3:40049 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Aug 06 00:58:40 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:40 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:40.403666 1489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:40.397516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.1:35413)
0806 00:58:40.397679 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:40.397692 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:40.397758 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:40.397919 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:40.397920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:40.398101 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:40.398158 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:40.398160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:40.398994 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:40.398996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:40.399657 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:40.399660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:40.400073 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:40.400074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:40.400125 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:40.400771 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:40.400777 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:40.403602 (+ 2825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
W20260806 00:58:40.403975 1490 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:40.397634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:35413 (local address 127.23.242.62:45237)
0806 00:58:40.397776 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:40.397778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:40.397793 (+ 15us) server_negotiation.cc:407] Connection header received
0806 00:58:40.397824 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:40.397825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:40.397838 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:40.397905 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:40.398323 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:40.398902 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:40.399771 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:40.399982 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:40.400047 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:40.403913 (+ 3866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.1:35413: BlockingRecv error: recv got EOF from 127.23.242.1:35413 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
Aug 06 00:58:40 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:40 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:40.542735 1491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:40.535802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.2:34003)
0806 00:58:40.536058 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:40.536099 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:40.536190 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:40.536392 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:40.536393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:40.536584 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:40.536636 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:40.536637 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:40.537737 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:40.537738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:40.538403 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:40.538406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:40.538899 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:40.538900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:40.538957 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:40.540471 (+ 1514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:40.540476 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:40.542551 (+ 2075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":186,"threads_started":1}
W20260806 00:58:40.542790 1490 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:40.535920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:34003 (local address 127.23.242.62:45237)
0806 00:58:40.536021 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:40.536023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:40.536211 (+ 188us) server_negotiation.cc:407] Connection header received
0806 00:58:40.536249 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:40.536250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:40.536265 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:40.536345 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:40.536728 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:40.537529 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:40.538522 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:40.538761 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:40.538827 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:40.542738 (+ 3911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.2:34003: BlockingRecv error: recv got EOF from 127.23.242.2:34003 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 06 00:58:40 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:40 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:40.671532 1492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:40.665816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.3:36545)
0806 00:58:40.666168 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:40.666181 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:40.666288 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:40.666712 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:40.666714 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:40.666898 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:40.666944 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:40.666946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:40.667617 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:40.667618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:40.668154 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:40.668155 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:40.668531 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:40.668532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:40.668595 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:40.669201 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:40.669206 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:40.671426 (+ 2220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":90,"threads_started":1}
W20260806 00:58:40.671591 1490 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:40.666280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:36545 (local address 127.23.242.62:45237)
0806 00:58:40.666378 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:40.666380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:40.666393 (+ 13us) server_negotiation.cc:407] Connection header received
0806 00:58:40.666543 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:40.666545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:40.666569 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:40.666690 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:40.667022 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:40.667531 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:40.668253 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:40.668446 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:40.668529 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:40.671522 (+ 2993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.3:36545: BlockingRecv error: recv got EOF from 127.23.242.3:36545 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 06 00:58:41 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:41 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:41.409822 1493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:41.404358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.1:58829)
0806 00:58:41.404593 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:41.404607 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:41.404684 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:41.404939 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:41.404939 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:41.405131 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:41.405184 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:41.405185 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:41.405951 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:41.405952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:41.406711 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:41.406714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:41.407145 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:41.407146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:41.407213 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:41.407754 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:41.407758 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:41.409727 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":144,"threads_started":1}
W20260806 00:58:41.409893 1494 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:41.404537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:58829 (local address 127.23.242.62:45237)
0806 00:58:41.404716 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:41.404718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:41.404731 (+ 13us) server_negotiation.cc:407] Connection header received
0806 00:58:41.404765 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:41.404766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:41.404781 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:41.404932 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:41.405310 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:41.405843 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:41.406818 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:41.407025 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:41.407098 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:41.409838 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.1:58829: BlockingRecv error: recv got EOF from 127.23.242.1:58829 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":99,"threads_started":1}
Aug 06 00:58:41 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:41 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:41.548979 1495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:41.543362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.2:58325)
0806 00:58:41.543518 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:41.543533 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:41.543628 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:41.543831 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:41.543833 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:41.544034 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:41.544085 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:41.544087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:41.545067 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:41.545069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:41.545723 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:41.545725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:41.546133 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:41.546134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:41.546215 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:41.546829 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:41.546834 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:41.548866 (+ 2032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
W20260806 00:58:41.549046 1494 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:41.543509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:58325 (local address 127.23.242.62:45237)
0806 00:58:41.543631 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:41.543634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:41.543715 (+ 81us) server_negotiation.cc:407] Connection header received
0806 00:58:41.543746 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:41.543747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:41.543759 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:41.543826 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:41.544180 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:41.544965 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:41.545816 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:41.546028 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:41.546133 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:41.548982 (+ 2849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.2:58325: BlockingRecv error: recv got EOF from 127.23.242.2:58325 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 06 00:58:41 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:41 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:41.678079 1496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:41.672453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.3:33421)
0806 00:58:41.672675 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:41.672687 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:41.672760 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:41.672901 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:41.672901 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:41.673077 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:41.673129 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:41.673131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:41.673832 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:41.673834 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:41.674430 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:41.674431 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:41.674928 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:41.674929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:41.674986 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:41.675764 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:41.675769 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:41.678013 (+ 2244us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":93,"threads_started":1}
W20260806 00:58:41.678357 1494 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:41.672509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:33421 (local address 127.23.242.62:45237)
0806 00:58:41.672628 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:41.672629 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:41.672766 (+ 137us) server_negotiation.cc:407] Connection header received
0806 00:58:41.672820 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:41.672821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:41.672834 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:41.672877 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:41.673208 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:41.673739 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:41.674536 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:41.674818 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:41.674885 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:41.678289 (+ 3404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.3:33421: BlockingRecv error: recv got EOF from 127.23.242.3:33421 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 06 00:58:42 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:42 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:42.421443 1497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:42.410458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.1:45133)
0806 00:58:42.410682 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:42.410694 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:42.410762 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:42.410901 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:42.410902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:42.411073 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:42.411125 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:42.411127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:42.411909 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:42.411910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:42.412463 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:42.412465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:42.413296 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:42.413297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:42.413355 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:42.418806 (+ 5451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:42.418812 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:42.421352 (+ 2540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":100,"threads_started":1}
W20260806 00:58:42.421641 1498 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:42.410495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:45133 (local address 127.23.242.62:45237)
0806 00:58:42.410687 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:42.410688 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:42.410781 (+ 93us) server_negotiation.cc:407] Connection header received
0806 00:58:42.410809 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:42.410809 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:42.410821 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:42.410883 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:42.411230 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:42.411802 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:42.412593 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:42.413130 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:42.413271 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:42.421560 (+ 8289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.1:45133: BlockingRecv error: recv got EOF from 127.23.242.1:45133 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":125,"threads_started":1}
Aug 06 00:58:42 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:42 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:42.556929 1499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:42.549710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.2:59203)
0806 00:58:42.549892 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:42.549938 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:42.550036 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:42.550241 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:42.550242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:42.550507 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:42.550639 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:42.550643 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:42.551766 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:42.551768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:42.552661 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:42.552666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:42.553237 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:42.553238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:42.553317 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:42.553971 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:42.553976 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:42.556823 (+ 2847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":96,"threads_started":1}
W20260806 00:58:42.557082 1498 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:42.549899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:59203 (local address 127.23.242.62:45237)
0806 00:58:42.549956 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:42.549959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:42.550011 (+ 52us) server_negotiation.cc:407] Connection header received
0806 00:58:42.550111 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:42.550112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:42.550131 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:42.550213 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:42.550731 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:42.551658 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:42.552768 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:42.553125 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:42.553213 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:42.556948 (+ 3735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.2:59203: BlockingRecv error: recv got EOF from 127.23.242.2:59203 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Aug 06 00:58:42 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:42 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:42.684604 1500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:42.678737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.3:56863)
0806 00:58:42.678903 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:42.678918 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:42.679006 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:42.679234 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:42.679235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:42.679434 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:42.679483 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:42.679485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:42.680238 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:42.680240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:42.680814 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:42.680815 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:42.681193 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:42.681194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:42.681256 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:42.681874 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:42.681880 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:42.684536 (+ 2656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":124,"threads_started":1}
W20260806 00:58:42.684800 1498 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:42.678910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:56863 (local address 127.23.242.62:45237)
0806 00:58:42.678981 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:42.678983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:42.678994 (+ 11us) server_negotiation.cc:407] Connection header received
0806 00:58:42.679130 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:42.679131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:42.679154 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:42.679227 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:42.679610 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:42.680139 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:42.680904 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:42.681115 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:42.681196 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:42.684664 (+ 3468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.3:56863: BlockingRecv error: recv got EOF from 127.23.242.3:56863 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 06 00:58:43 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:43 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:43.428184 1501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:43.422236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.1:50579)
0806 00:58:43.422452 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:43.422470 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:43.422547 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:43.423127 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:43.423128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:43.423324 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:43.423379 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:43.423380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:43.424126 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:43.424127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:43.424670 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:43.424672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:43.425108 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:43.425109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:43.425154 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:43.425848 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:43.425852 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:43.428108 (+ 2256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":120,"threads_started":1}
W20260806 00:58:43.428429 1502 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:43.422494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:50579 (local address 127.23.242.62:45237)
0806 00:58:43.422808 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:43.422810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:43.422822 (+ 12us) server_negotiation.cc:407] Connection header received
0806 00:58:43.422849 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:43.422849 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:43.422861 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:43.422933 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:43.423492 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:43.424017 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:43.424800 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:43.424995 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:43.425055 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:43.428331 (+ 3276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.1:50579: BlockingRecv error: recv got EOF from 127.23.242.1:50579 (error 108)
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":221,"threads_started":1}
Aug 06 00:58:43 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:43 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:43.563351 1503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:43.557475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.2:49091)
0806 00:58:43.557702 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:43.557716 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:43.557830 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:43.558034 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:43.558035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:43.558241 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:43.558293 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:43.558295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:43.559126 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:43.559127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:43.559718 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:43.559720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:43.560150 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:43.560152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:43.560224 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:43.560819 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:43.560833 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:43.563235 (+ 2402us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":87,"threads_started":1}
W20260806 00:58:43.563411 1502 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:43.557586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:49091 (local address 127.23.242.62:45237)
0806 00:58:43.557701 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:43.557703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:43.557814 (+ 111us) server_negotiation.cc:407] Connection header received
0806 00:58:43.557924 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:43.557925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:43.557941 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:43.558027 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:43.558404 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:43.559030 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:43.559818 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:43.560031 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:43.560139 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:43.563336 (+ 3197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.2:49091: BlockingRecv error: recv got EOF from 127.23.242.2:49091 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 06 00:58:43 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:43 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:43.691177 1504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:43.685168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.3:56217)
0806 00:58:43.685353 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:43.685364 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:43.685412 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:43.685556 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:43.685557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:43.685729 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:43.685786 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:43.685788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:43.686649 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:43.686651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:43.687461 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:43.687465 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:43.687914 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:43.687915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:43.687980 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:43.688910 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:43.688916 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:43.691108 (+ 2192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":82,"threads_started":1}
W20260806 00:58:43.691352 1502 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:43.685344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:56217 (local address 127.23.242.62:45237)
0806 00:58:43.685417 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:43.685420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:43.685432 (+ 12us) server_negotiation.cc:407] Connection header received
0806 00:58:43.685463 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:43.685464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:43.685478 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:43.685537 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:43.685884 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:43.686429 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:43.687580 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:43.687788 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:43.687863 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:43.691221 (+ 3358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.3:56217: BlockingRecv error: recv got EOF from 127.23.242.3:56217 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 06 00:58:44 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:44 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:44.435338 1505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:44.428885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.1:49797)
0806 00:58:44.429067 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:44.429092 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:44.429169 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:44.429648 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:44.429649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:44.429834 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:44.429890 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:44.429892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:44.430769 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:44.430770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:44.431382 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:44.431383 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:44.432108 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:44.432110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:44.432188 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:44.432861 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:44.432868 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:44.435258 (+ 2390us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":89,"threads_started":1}
W20260806 00:58:44.435509 1506 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:44.429003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:49797 (local address 127.23.242.62:45237)
0806 00:58:44.429372 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:44.429375 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:44.429395 (+ 20us) server_negotiation.cc:407] Connection header received
0806 00:58:44.429444 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:44.429445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:44.429464 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:44.429582 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:44.430032 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:44.430646 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:44.431504 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:44.431961 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:44.432034 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:44.435414 (+ 3380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.1:49797: BlockingRecv error: recv got EOF from 127.23.242.1:49797 (error 108)
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":127,"threads_started":1}
Aug 06 00:58:44 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:44 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:44.569950 1507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:44.564117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.2:43139)
0806 00:58:44.564314 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:44.564326 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:44.564389 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:44.564566 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:44.564566 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:44.564761 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:44.564824 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:44.564825 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:44.565561 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:44.565562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:44.566166 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:44.566168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:44.566650 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:44.566651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:44.566705 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:44.567252 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:44.567257 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:44.569853 (+ 2596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":124,"threads_started":1}
W20260806 00:58:44.570199 1506 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:44.564267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:43139 (local address 127.23.242.62:45237)
0806 00:58:44.564348 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:44.564350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:44.564373 (+ 23us) server_negotiation.cc:407] Connection header received
0806 00:58:44.564459 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:44.564459 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:44.564472 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:44.564543 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:44.564895 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:44.565461 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:44.566277 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:44.566496 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:44.566560 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:44.570093 (+ 3533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.2:43139: BlockingRecv error: recv got EOF from 127.23.242.2:43139 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 06 00:58:44 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:44 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:44.698153 1508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:44.691901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.3:54063)
0806 00:58:44.692222 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:44.692234 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:44.692319 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:44.692564 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:44.692566 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:44.692821 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:44.692876 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:44.692878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:44.693610 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:44.693611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:44.694245 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:44.694248 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:44.694850 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:44.694851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:44.694902 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:44.695633 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:44.695640 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:44.698071 (+ 2431us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":176,"threads_started":1}
W20260806 00:58:44.698369 1506 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:44.692080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:54063 (local address 127.23.242.62:45237)
0806 00:58:44.692185 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:44.692187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:44.692326 (+ 139us) server_negotiation.cc:407] Connection header received
0806 00:58:44.692403 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:44.692404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:44.692423 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:44.692481 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:44.692978 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:44.693531 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:44.694441 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:44.694735 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:44.694811 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:44.698182 (+ 3371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.3:54063: BlockingRecv error: recv got EOF from 127.23.242.3:54063 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 06 00:58:45 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:45 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:45.441334 1509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:45.436005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.1:49609)
0806 00:58:45.436180 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:45.436193 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:45.436255 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:45.436456 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:45.436458 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:45.436695 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:45.436748 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:45.436750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:45.437513 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:45.437514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:45.438061 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:45.438063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:45.438455 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:45.438456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:45.438510 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:45.439156 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:45.439161 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:45.441262 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":118,"threads_started":1}
W20260806 00:58:45.441586 1511 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:45.436105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:49609 (local address 127.23.242.62:45237)
0806 00:58:45.436265 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:45.436267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:45.436277 (+ 10us) server_negotiation.cc:407] Connection header received
0806 00:58:45.436300 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:45.436301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:45.436311 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:45.436380 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:45.436879 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:45.437411 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:45.438159 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:45.438352 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:45.438423 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:45.441518 (+ 3095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.1:49609: BlockingRecv error: recv got EOF from 127.23.242.1:49609 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
Aug 06 00:58:45 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:45 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:45.576757 1512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:45.570686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.2:59645)
0806 00:58:45.570934 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:45.570954 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:45.571023 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:45.571229 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:45.571230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:45.571664 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:45.571727 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:45.571730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:45.572716 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:45.572717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:45.573373 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:45.573376 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:45.573869 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:45.573872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:45.573963 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:45.574705 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:45.574710 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:45.576684 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":108,"threads_started":1}
W20260806 00:58:45.577077 1511 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:45.570834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:59645 (local address 127.23.242.62:45237)
0806 00:58:45.570900 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:45.570903 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:45.571028 (+ 125us) server_negotiation.cc:407] Connection header received
0806 00:58:45.571072 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:45.571073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:45.571087 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:45.571153 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:45.571860 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:45.572587 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:45.573495 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:45.573689 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:45.573789 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:45.577007 (+ 3218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.2:59645: BlockingRecv error: recv got EOF from 127.23.242.2:59645 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 06 00:58:45 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:45 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:45.704978 1513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:45.698970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.3:49397)
0806 00:58:45.699133 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:45.699147 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:45.699209 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:45.699835 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:45.699837 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:45.700116 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:45.700166 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:45.700169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:45.701201 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:45.701202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:45.701818 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:45.701820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:45.702226 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:45.702227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:45.702275 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:45.702904 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:45.702909 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:45.704911 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":99,"threads_started":1}
W20260806 00:58:45.705209 1511 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:45.699104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:49397 (local address 127.23.242.62:45237)
0806 00:58:45.699195 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:45.699197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:45.699214 (+ 17us) server_negotiation.cc:407] Connection header received
0806 00:58:45.699669 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:45.699670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:45.699689 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:45.699770 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:45.700289 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:45.701091 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:45.701936 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:45.702134 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:45.702201 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:45.705092 (+ 2891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.3:49397: BlockingRecv error: recv got EOF from 127.23.242.3:49397 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 06 00:58:46 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:46 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:46.447806 1514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:46.442009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.1:51245)
0806 00:58:46.442228 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:46.442241 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:46.442307 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:46.442512 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:46.442513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:46.442725 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:46.442781 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:46.442783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:46.443533 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:46.443534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:46.444186 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:46.444189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:46.444764 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:46.444766 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:46.444820 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:46.445511 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:46.445516 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:46.447701 (+ 2185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":112,"threads_started":1}
W20260806 00:58:46.447903 1515 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:46.442150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:51245 (local address 127.23.242.62:45237)
0806 00:58:46.442292 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:46.442294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:46.442307 (+ 13us) server_negotiation.cc:407] Connection header received
0806 00:58:46.442373 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:46.442374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:46.442385 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:46.442438 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:46.442889 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:46.443435 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:46.444358 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:46.444625 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:46.444737 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:46.447825 (+ 3088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.1:51245: BlockingRecv error: recv got EOF from 127.23.242.1:51245 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
Aug 06 00:58:46 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:46 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:46.582839 1516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:46.577383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.2:49529)
0806 00:58:46.577538 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:46.577576 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:46.577659 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:46.577883 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:46.577884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:46.578092 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:46.578143 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:46.578145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:46.579034 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:46.579035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:46.579764 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:46.579766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:46.580217 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:46.580218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:46.580295 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:46.580837 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:46.580842 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:46.582722 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
W20260806 00:58:46.582875 1515 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:46.577539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:49529 (local address 127.23.242.62:45237)
0806 00:58:46.577618 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:46.577621 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:46.577636 (+ 15us) server_negotiation.cc:407] Connection header received
0806 00:58:46.577790 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:46.577791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:46.577803 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:46.577874 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:46.578238 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:46.578933 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:46.579863 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:46.580126 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:46.580210 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:46.582806 (+ 2596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.2:49529: BlockingRecv error: recv got EOF from 127.23.242.2:49529 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 06 00:58:46 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:46 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:46.710901 1517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:46.705484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.3:45737)
0806 00:58:46.705710 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:46.705726 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:46.705811 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:46.705991 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:46.705992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:46.706277 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:46.706325 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:46.706327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:46.707141 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:46.707142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:46.707750 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:46.707752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:46.708158 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:46.708159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:46.708221 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:46.708784 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:46.708790 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:46.710822 (+ 2032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":119,"threads_started":1}
W20260806 00:58:46.711066 1515 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:46.705579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:45737 (local address 127.23.242.62:45237)
0806 00:58:46.705715 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:46.705718 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:46.705803 (+ 85us) server_negotiation.cc:407] Connection header received
0806 00:58:46.705875 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:46.705876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:46.705893 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:46.705974 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:46.706420 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:46.707046 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:46.707846 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:46.708066 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:46.708152 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:46.710961 (+ 2809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.3:45737: BlockingRecv error: recv got EOF from 127.23.242.3:45737 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Aug 06 00:58:47 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:47 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:47.463757 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:47.448362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.1:57529)
0806 00:58:47.448519 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:47.448532 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:47.448609 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:47.448754 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:47.448755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:47.448924 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:47.448973 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:47.448975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:47.449755 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:47.449757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:47.450437 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:47.450439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:47.451046 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:47.451048 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:47.451115 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:47.460730 (+ 9615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:47.460738 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:47.463673 (+ 2935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
W20260806 00:58:47.464128 1519 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:47.448470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:57529 (local address 127.23.242.62:45237)
0806 00:58:47.448629 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:47.448632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:47.448644 (+ 12us) server_negotiation.cc:407] Connection header received
0806 00:58:47.448668 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:47.448669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:47.448680 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:47.448749 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:47.449061 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:47.449589 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:47.450636 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:47.450918 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:47.450999 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:47.464042 (+ 13043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.1:57529: BlockingRecv error: recv got EOF from 127.23.242.1:57529 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":74,"threads_started":1}
Aug 06 00:58:47 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:47 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:47.589564 1520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:47.583500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.2:55359)
0806 00:58:47.583649 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:47.583664 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:47.583738 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:47.584009 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:47.584011 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:47.584299 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:47.584379 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:47.584384 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:47.585199 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:47.585200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:47.585789 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:47.585791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:47.586464 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:47.586465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:47.586515 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:47.587089 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:47.587094 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:47.589499 (+ 2405us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
W20260806 00:58:47.589716 1519 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:47.583574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:55359 (local address 127.23.242.62:45237)
0806 00:58:47.583662 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:47.583664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:47.583741 (+ 77us) server_negotiation.cc:407] Connection header received
0806 00:58:47.583797 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:47.583798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:47.583809 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:47.583878 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:47.584548 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:47.585069 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:47.585967 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:47.586219 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:47.586457 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:47.589618 (+ 3161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.2:55359: BlockingRecv error: recv got EOF from 127.23.242.2:55359 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 06 00:58:47 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:47 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.3@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:47.717581 1521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:47.711501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.3:36069)
0806 00:58:47.711749 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:47.711762 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:47.711839 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:47.712004 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:47.712005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:47.712204 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:47.712252 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:47.712254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:47.713197 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:47.713198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:47.713805 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:47.713807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:47.714223 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:47.714224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:47.714294 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:47.714997 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:47.715002 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:47.717463 (+ 2461us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":193,"threads_started":1}
W20260806 00:58:47.717640 1519 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:47.711742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:36069 (local address 127.23.242.62:45237)
0806 00:58:47.711843 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:47.711845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:47.711878 (+ 33us) server_negotiation.cc:407] Connection header received
0806 00:58:47.711909 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:47.711910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:47.711933 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:47.711998 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:47.712354 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:47.712990 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:47.713905 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:47.714121 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:47.714222 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:47.717562 (+ 3340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.3:36069: BlockingRecv error: recv got EOF from 127.23.242.3:36069 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 06 00:58:48 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:48 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.1@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:48.469521 1522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:48.464521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.1:33435)
0806 00:58:48.464680 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:48.464693 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:48.464761 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:48.464987 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:48.464988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:48.465179 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:48.465229 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:48.465231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:48.466007 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:48.466008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:48.466651 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:48.466653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:48.467067 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:48.467068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:48.467116 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:48.467682 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:48.467687 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:48.469460 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":107,"threads_started":1}
W20260806 00:58:48.469774 1523 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:48.464618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:33435 (local address 127.23.242.62:45237)
0806 00:58:48.464825 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:48.464827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:48.464837 (+ 10us) server_negotiation.cc:407] Connection header received
0806 00:58:48.464863 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:48.464864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:48.464877 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:48.464952 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:48.465325 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:48.465912 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:48.466761 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:48.466972 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:48.467042 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:48.469628 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.1:33435: BlockingRecv error: recv got EOF from 127.23.242.1:33435 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":135,"threads_started":1}
Aug 06 00:58:48 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 00:58:48 dist-test-slave-dhph krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.242.2@KRBTEST.COM for oryx/127.23.242.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 00:58:48.595549 1524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:48.590157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45237 (local address 127.23.242.2:33097)
0806 00:58:48.590324 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:48.590335 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:48.590395 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:48.590533 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:48.590533 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:48.590728 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:48.590778 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:48.590780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:48.591488 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:48.591489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:48.592061 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:48.592063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:48.592460 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:48.592461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:48.592517 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:48.593065 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:48.593070 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:48.595469 (+ 2399us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.242.62:45237: Server oryx/127.23.242.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":65,"threads_started":1}
W20260806 00:58:48.595767 1523 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:48.590321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:33097 (local address 127.23.242.62:45237)
0806 00:58:48.590392 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:48.590394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:48.590404 (+ 10us) server_negotiation.cc:407] Connection header received
0806 00:58:48.590447 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:48.590448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:48.590459 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:48.590510 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:48.590848 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:48.591389 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:48.592175 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:48.592380 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:48.592461 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:48.595702 (+ 3241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.242.2:33097: BlockingRecv error: recv got EOF from 127.23.242.2:33097 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 00:58:48.617390 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 1040
I20260806 00:58:48.626914 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 1196
I20260806 00:58:48.632900 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 1330
I20260806 00:58:48.639374 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 967
2026-08-06T00:58:48Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17831 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T00:58:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T00:58:48Z Disabled control of system clock
I20260806 00:58:48.675084 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:33633
--webserver_interface=127.23.242.62
--webserver_port=0
--builtin_ntp_servers=127.23.242.20:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:33633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 00:58:48.771919 1528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:48.772171 1528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:48.772197 1528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:48.773375 1528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 00:58:48.773423 1528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:48.773438 1528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:48.773453 1528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 00:58:48.773468 1528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 00:58:48.774852 1528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:33633
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:33633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:48.775082 1528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:48.775367 1528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:48.783246 1536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:48.783252 1534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:48.783291 1528 server_base.cc:1034] running on GCE node
W20260806 00:58:48.783252 1533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:48.783843 1528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:48.784198 1528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:48.785372 1528 hybrid_clock.cc:648] HybridClock initialized: now 1785977928785392 us; error 96 us; skew 500 ppm
I20260806 00:58:48.787629 1528 webserver.cc:460] Webserver started at http://127.23.242.62:40321/ using document root <none> and password file <none>
I20260806 00:58:48.787984 1528 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:48.788060 1528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:48.788208 1528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:48.789893 1528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "72dece63fbbb420b89f2e14ec935f267"
format_stamp: "Formatted at 2026-08-06 00:58:48 on dist-test-slave-dhph"
server_key: "fde5ed0af13a0a5e33c613390815b23b"
server_key_iv: "c81c59ff09cdbc9f024bdcc6584377d2"
server_key_version: "encryptionkey@0"
I20260806 00:58:48.790431 1528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "72dece63fbbb420b89f2e14ec935f267"
format_stamp: "Formatted at 2026-08-06 00:58:48 on dist-test-slave-dhph"
server_key: "fde5ed0af13a0a5e33c613390815b23b"
server_key_iv: "c81c59ff09cdbc9f024bdcc6584377d2"
server_key_version: "encryptionkey@0"
I20260806 00:58:48.791996 1528 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:48.793148 1542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:48.793565 1528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:48.793753 1528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "72dece63fbbb420b89f2e14ec935f267"
format_stamp: "Formatted at 2026-08-06 00:58:48 on dist-test-slave-dhph"
server_key: "fde5ed0af13a0a5e33c613390815b23b"
server_key_iv: "c81c59ff09cdbc9f024bdcc6584377d2"
server_key_version: "encryptionkey@0"
I20260806 00:58:48.793855 1528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:48.810539 1528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:48.810957 1528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:48.811112 1528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:48.815596 1528 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:33633
I20260806 00:58:48.815631 1595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:33633 every 8 connection(s)
I20260806 00:58:48.816051 1528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 00:58:48.816807 1596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:48.820251 1596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72dece63fbbb420b89f2e14ec935f267: Bootstrap starting.
I20260806 00:58:48.820950 1596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72dece63fbbb420b89f2e14ec935f267: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:48.821406 1596 log.cc:824] T 00000000000000000000000000000000 P 72dece63fbbb420b89f2e14ec935f267: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:48.821954 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 1528
I20260806 00:58:48.822077 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 00:58:48.822324 24520 external_mini_cluster.cc:1428] Setting key d7cfc720db10207419ec3913223f9811
I20260806 00:58:48.822784 1596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72dece63fbbb420b89f2e14ec935f267: No bootstrap required, opened a new log
I20260806 00:58:48.825263 1596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72dece63fbbb420b89f2e14ec935f267 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72dece63fbbb420b89f2e14ec935f267" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 33633 } }
I20260806 00:58:48.825534 1596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72dece63fbbb420b89f2e14ec935f267 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:48.825582 1596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72dece63fbbb420b89f2e14ec935f267 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72dece63fbbb420b89f2e14ec935f267, State: Initialized, Role: FOLLOWER
I20260806 00:58:48.825774 1596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72dece63fbbb420b89f2e14ec935f267 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72dece63fbbb420b89f2e14ec935f267" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 33633 } }
I20260806 00:58:48.825891 1596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72dece63fbbb420b89f2e14ec935f267 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 00:58:48.825951 1596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72dece63fbbb420b89f2e14ec935f267 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 00:58:48.826004 1596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72dece63fbbb420b89f2e14ec935f267 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:48.827595 1596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72dece63fbbb420b89f2e14ec935f267 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72dece63fbbb420b89f2e14ec935f267" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 33633 } }
I20260806 00:58:48.827778 1596 leader_election.cc:302] T 00000000000000000000000000000000 P 72dece63fbbb420b89f2e14ec935f267 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72dece63fbbb420b89f2e14ec935f267; no voters:
I20260806 00:58:48.827975 1596 leader_election.cc:288] T 00000000000000000000000000000000 P 72dece63fbbb420b89f2e14ec935f267 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 00:58:48.828119 1601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72dece63fbbb420b89f2e14ec935f267 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:48.828297 1596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72dece63fbbb420b89f2e14ec935f267 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 00:58:48.828349 1601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72dece63fbbb420b89f2e14ec935f267 [term 1 LEADER]: Becoming Leader. State: Replica: 72dece63fbbb420b89f2e14ec935f267, State: Running, Role: LEADER
I20260806 00:58:48.828495 1601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72dece63fbbb420b89f2e14ec935f267 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72dece63fbbb420b89f2e14ec935f267" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 33633 } }
I20260806 00:58:48.828975 1603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72dece63fbbb420b89f2e14ec935f267 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72dece63fbbb420b89f2e14ec935f267" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72dece63fbbb420b89f2e14ec935f267" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 33633 } } }
I20260806 00:58:48.828958 1601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72dece63fbbb420b89f2e14ec935f267 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72dece63fbbb420b89f2e14ec935f267. Latest consensus state: current_term: 1 leader_uuid: "72dece63fbbb420b89f2e14ec935f267" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72dece63fbbb420b89f2e14ec935f267" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 33633 } } }
I20260806 00:58:48.829105 1603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72dece63fbbb420b89f2e14ec935f267 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:48.829314 1601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72dece63fbbb420b89f2e14ec935f267 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:48.829560 1610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 00:58:48.830175 1610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 00:58:48.832453 1610 catalog_manager.cc:1347] Generated new cluster ID: 09f954a97e8741079bbe9b63c9235f49
I20260806 00:58:48.832490 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:48.823055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33670 (local address 127.23.242.62:33633)
0806 00:58:48.823362 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:48.823371 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:48.823399 (+ 28us) server_negotiation.cc:407] Connection header received
0806 00:58:48.825258 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:48.825276 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:48.825363 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:48.825556 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:48.825963 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:48.828504 (+ 2541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:48.829693 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:48.830362 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:48.830892 (+ 530us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:48.831490 (+ 598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:48.831494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:48.831494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 00:58:48.831726 (+ 232us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:48.831775 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 00:58:48.831791 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:48.831873 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:48.832153 (+ 280us) server_negotiation.cc:299] Negotiation successful
0806 00:58:48.832245 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":129,"threads_started":1}
I20260806 00:58:48.832517 1610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 00:58:48.856932 1610 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 00:58:48.857700 1610 catalog_manager.cc:1504] Loading token signing keys...
I20260806 00:58:48.866468 1610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72dece63fbbb420b89f2e14ec935f267: Generated new TSK 0
I20260806 00:58:48.867209 1610 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 00:58:48.870386 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:0
--local_ip_for_outbound_sockets=127.23.242.1
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:33633
--builtin_ntp_servers=127.23.242.20:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 00:58:48.982856 1620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:48.983116 1620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:48.983160 1620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:48.984745 1620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:48.984839 1620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:48.984898 1620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 00:58:48.987090 1620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:33633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:48.987510 1620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:48.987869 1620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:48.994750 1625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:48.994750 1628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:48.994992 1626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:48.995639 1620 server_base.cc:1034] running on GCE node
I20260806 00:58:48.996030 1620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:48.996407 1620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:48.997646 1620 hybrid_clock.cc:648] HybridClock initialized: now 1785977928997644 us; error 66 us; skew 500 ppm
I20260806 00:58:48.999917 1620 webserver.cc:460] Webserver started at http://127.23.242.1:35673/ using document root <none> and password file <none>
I20260806 00:58:49.000494 1620 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:49.000617 1620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:49.000844 1620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:49.005801 1620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/data/instance:
uuid: "1feefde8c160404382a201946a38c2ed"
format_stamp: "Formatted at 2026-08-06 00:58:48 on dist-test-slave-dhph"
server_key: "1f7058514e5bdeaa0fcbcb8a4ac96865"
server_key_iv: "7044e33d2937a84c1d18c116a22184c3"
server_key_version: "encryptionkey@0"
I20260806 00:58:49.007158 1620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance:
uuid: "1feefde8c160404382a201946a38c2ed"
format_stamp: "Formatted at 2026-08-06 00:58:48 on dist-test-slave-dhph"
server_key: "1f7058514e5bdeaa0fcbcb8a4ac96865"
server_key_iv: "7044e33d2937a84c1d18c116a22184c3"
server_key_version: "encryptionkey@0"
I20260806 00:58:49.009212 1620 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 00:58:49.010893 1634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:49.011173 1620 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:49.011277 1620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "1feefde8c160404382a201946a38c2ed"
format_stamp: "Formatted at 2026-08-06 00:58:48 on dist-test-slave-dhph"
server_key: "1f7058514e5bdeaa0fcbcb8a4ac96865"
server_key_iv: "7044e33d2937a84c1d18c116a22184c3"
server_key_version: "encryptionkey@0"
I20260806 00:58:49.011343 1620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:49.021020 1620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:49.021366 1620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:49.021481 1620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:49.021720 1620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:49.022149 1620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:49.022192 1620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:49.022219 1620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:49.022234 1620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:49.032080 1620 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:46735
I20260806 00:58:49.032186 1750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:46735 every 8 connection(s)
I20260806 00:58:49.032531 1620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 00:58:49.038475 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:49.033541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:46653 (local address 127.23.242.62:33633)
0806 00:58:49.033682 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:49.033684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:49.033892 (+ 208us) server_negotiation.cc:407] Connection header received
0806 00:58:49.034511 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:49.034512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:49.034528 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:49.034619 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:49.035355 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:49.035894 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:49.036905 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:49.037125 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:49.037208 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:49.038175 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:49.038176 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:49.038177 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 00:58:49.038183 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:49.038199 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 00:58:49.038204 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:49.038253 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:49.038351 (+ 98us) server_negotiation.cc:299] Negotiation successful
0806 00:58:49.038386 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260806 00:58:49.038915 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:49.033534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:33633 (local address 127.23.242.1:46653)
0806 00:58:49.033764 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:49.033800 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:49.034395 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:49.034699 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:49.034702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:49.034710 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:49.035244 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:49.035248 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:49.036002 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:49.036003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:49.036805 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:49.036807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:49.037223 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:49.037224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:49.037363 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:49.038018 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 00:58:49.038019 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:49.038048 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 00:58:49.038072 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:49.038253 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:49.038259 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:49.038262 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 00:58:49.038355 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 00:58:49.038466 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":94,"threads_started":1}
I20260806 00:58:49.039781 1751 heartbeater.cc:347] Connected to a master server at 127.23.242.62:33633
I20260806 00:58:49.040016 1751 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:49.040355 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 1620
I20260806 00:58:49.040396 1751 heartbeater.cc:511] Master 127.23.242.62:33633 requested a full tablet report, sending...
I20260806 00:58:49.040453 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance
I20260806 00:58:49.040756 24520 external_mini_cluster.cc:1428] Setting key 355a727b6471f48025e1e1a060e3424f
I20260806 00:58:49.041285 1559 ts_manager.cc:194] Registered new tserver with Master: 1feefde8c160404382a201946a38c2ed (127.23.242.1:46735)
I20260806 00:58:49.042196 1559 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.242.1:46653
I20260806 00:58:49.042227 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:0
--local_ip_for_outbound_sockets=127.23.242.2
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:33633
--builtin_ntp_servers=127.23.242.20:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 00:58:49.132181 1755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:49.132431 1755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:49.132469 1755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:49.133642 1755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:49.133709 1755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:49.133769 1755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 00:58:49.135340 1755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:33633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:49.135645 1755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:49.135994 1755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:49.141438 1760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:49.141399 1763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:49.141563 1755 server_base.cc:1034] running on GCE node
W20260806 00:58:49.141541 1761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:49.141969 1755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:49.142228 1755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:49.143407 1755 hybrid_clock.cc:648] HybridClock initialized: now 1785977929143394 us; error 29 us; skew 500 ppm
I20260806 00:58:49.144737 1755 webserver.cc:460] Webserver started at http://127.23.242.2:39547/ using document root <none> and password file <none>
I20260806 00:58:49.145015 1755 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:49.145089 1755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:49.145277 1755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:49.146390 1755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/data/instance:
uuid: "b4b51a45a95a41a3914a940f14e9e828"
format_stamp: "Formatted at 2026-08-06 00:58:49 on dist-test-slave-dhph"
server_key: "edecd2572e5c39f1d94d1f11254a598a"
server_key_iv: "f4e98f1603cfe7dd03c654ecbf55304a"
server_key_version: "encryptionkey@0"
I20260806 00:58:49.146806 1755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance:
uuid: "b4b51a45a95a41a3914a940f14e9e828"
format_stamp: "Formatted at 2026-08-06 00:58:49 on dist-test-slave-dhph"
server_key: "edecd2572e5c39f1d94d1f11254a598a"
server_key_iv: "f4e98f1603cfe7dd03c654ecbf55304a"
server_key_version: "encryptionkey@0"
I20260806 00:58:49.148403 1755 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:49.149344 1769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:49.149611 1755 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:49.149735 1755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "b4b51a45a95a41a3914a940f14e9e828"
format_stamp: "Formatted at 2026-08-06 00:58:49 on dist-test-slave-dhph"
server_key: "edecd2572e5c39f1d94d1f11254a598a"
server_key_iv: "f4e98f1603cfe7dd03c654ecbf55304a"
server_key_version: "encryptionkey@0"
I20260806 00:58:49.149820 1755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:49.170667 1755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:49.171049 1755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:49.171221 1755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:49.171486 1755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:49.171867 1755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:49.171926 1755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:49.171969 1755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:49.172011 1755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:49.179277 1755 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:33807
I20260806 00:58:49.179328 1883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:33807 every 8 connection(s)
I20260806 00:58:49.179894 1755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 00:58:49.187270 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:49.181448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:44551 (local address 127.23.242.62:33633)
0806 00:58:49.181539 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:49.181542 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:49.181631 (+ 89us) server_negotiation.cc:407] Connection header received
0806 00:58:49.182747 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:49.182751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:49.182770 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:49.182856 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:49.183803 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:49.184306 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:49.185447 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:49.185638 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:49.185727 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:49.186699 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:49.186700 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:49.186701 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 00:58:49.186710 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:49.186732 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 00:58:49.186737 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:49.186789 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:49.187175 (+ 386us) server_negotiation.cc:299] Negotiation successful
0806 00:58:49.187203 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 00:58:49.187654 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:49.181221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:33633 (local address 127.23.242.2:44551)
0806 00:58:49.181477 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:49.181533 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:49.182457 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:49.182971 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:49.182976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:49.182990 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:49.183668 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:49.183673 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:49.184434 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:49.184435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:49.185341 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:49.185344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:49.185743 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:49.185744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:49.185857 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:49.186446 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 00:58:49.186447 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:49.186479 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 00:58:49.186504 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:49.186864 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:49.186878 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:49.186883 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 00:58:49.187023 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 00:58:49.187178 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":82,"threads_started":1}
I20260806 00:58:49.188349 1884 heartbeater.cc:347] Connected to a master server at 127.23.242.62:33633
I20260806 00:58:49.188444 1884 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:49.188762 1884 heartbeater.cc:511] Master 127.23.242.62:33633 requested a full tablet report, sending...
I20260806 00:58:49.189250 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 1755
I20260806 00:58:49.189270 1559 ts_manager.cc:194] Registered new tserver with Master: b4b51a45a95a41a3914a940f14e9e828 (127.23.242.2:33807)
I20260806 00:58:49.189365 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance
I20260806 00:58:49.189576 24520 external_mini_cluster.cc:1428] Setting key c7c6f87d047613dbf367353b0f6073a0
I20260806 00:58:49.189834 1559 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.242.2:44551
I20260806 00:58:49.190653 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:0
--local_ip_for_outbound_sockets=127.23.242.3
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:33633
--builtin_ntp_servers=127.23.242.20:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 00:58:49.285377 1887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:49.285619 1887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:49.285653 1887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:49.287088 1887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:49.287140 1887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:49.287169 1887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 00:58:49.288592 1887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:33633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:49.288887 1887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:49.289232 1887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:49.295539 1893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:49.295606 1892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:49.295619 1895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:49.296497 1887 server_base.cc:1034] running on GCE node
I20260806 00:58:49.296828 1887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:49.297173 1887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:49.298663 1887 hybrid_clock.cc:648] HybridClock initialized: now 1785977929298539 us; error 127 us; skew 500 ppm
I20260806 00:58:49.300369 1887 webserver.cc:460] Webserver started at http://127.23.242.3:40619/ using document root <none> and password file <none>
I20260806 00:58:49.300668 1887 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:49.300751 1887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:49.300978 1887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:49.302332 1887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/data/instance:
uuid: "99bc39e235a9415a8282f19c64388276"
format_stamp: "Formatted at 2026-08-06 00:58:49 on dist-test-slave-dhph"
server_key: "8b4945bf9f883df3a9636ed3acd6c8e7"
server_key_iv: "ad4af56c083ac011af07b0389ba5acc5"
server_key_version: "encryptionkey@0"
I20260806 00:58:49.303153 1887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance:
uuid: "99bc39e235a9415a8282f19c64388276"
format_stamp: "Formatted at 2026-08-06 00:58:49 on dist-test-slave-dhph"
server_key: "8b4945bf9f883df3a9636ed3acd6c8e7"
server_key_iv: "ad4af56c083ac011af07b0389ba5acc5"
server_key_version: "encryptionkey@0"
I20260806 00:58:49.304688 1887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:49.305641 1901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:49.305907 1887 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:49.306013 1887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "99bc39e235a9415a8282f19c64388276"
format_stamp: "Formatted at 2026-08-06 00:58:49 on dist-test-slave-dhph"
server_key: "8b4945bf9f883df3a9636ed3acd6c8e7"
server_key_iv: "ad4af56c083ac011af07b0389ba5acc5"
server_key_version: "encryptionkey@0"
I20260806 00:58:49.306088 1887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:49.321976 1887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:49.322790 1887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:49.323040 1887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:49.323371 1887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:49.323814 1887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:49.323885 1887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:49.323941 1887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:49.324003 1887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:49.332010 1887 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:42715
I20260806 00:58:49.332060 2015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:42715 every 8 connection(s)
I20260806 00:58:49.332473 1887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
I20260806 00:58:49.336994 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 1887
I20260806 00:58:49.337100 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance
I20260806 00:58:49.337311 24520 external_mini_cluster.cc:1428] Setting key a1636f95b5a217d9834944f986fce2cd
I20260806 00:58:49.339978 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:49.333405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:54447 (local address 127.23.242.62:33633)
0806 00:58:49.333470 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:49.333472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:49.333606 (+ 134us) server_negotiation.cc:407] Connection header received
0806 00:58:49.334228 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:49.334230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:49.334243 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:49.334315 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:49.335337 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:49.336249 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:49.337695 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:49.337917 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:49.338012 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:49.339430 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:49.339433 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:49.339433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 00:58:49.339443 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:49.339478 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 00:58:49.339487 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:49.339593 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:49.339859 (+ 266us) server_negotiation.cc:299] Negotiation successful
0806 00:58:49.339903 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260806 00:58:49.340405 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:49.333252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:33633 (local address 127.23.242.3:54447)
0806 00:58:49.333464 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:49.333513 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:49.334068 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:49.334401 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:49.334405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:49.334414 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:49.335181 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:49.335187 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:49.336346 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:49.336347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:49.337604 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:49.337607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:49.338038 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:49.338039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:49.338224 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:49.339229 (+ 1005us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 00:58:49.339229 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:49.339262 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 00:58:49.339287 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:49.339650 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:49.339666 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:49.339671 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 00:58:49.339774 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 00:58:49.340016 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
I20260806 00:58:49.341387 2016 heartbeater.cc:347] Connected to a master server at 127.23.242.62:33633
I20260806 00:58:49.341502 2016 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:49.341682 2016 heartbeater.cc:511] Master 127.23.242.62:33633 requested a full tablet report, sending...
I20260806 00:58:49.342298 1558 ts_manager.cc:194] Registered new tserver with Master: 99bc39e235a9415a8282f19c64388276 (127.23.242.3:42715)
I20260806 00:58:49.342962 1558 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.242.3:54447
I20260806 00:58:49.350261 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 00:58:49.352865 1600 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:49.352017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33686 (local address 127.23.242.62:33633)
0806 00:58:49.352143 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:49.352145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:49.352337 (+ 192us) server_negotiation.cc:407] Connection header received
0806 00:58:49.352366 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:49.352367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:49.352380 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:49.352434 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:49.352625 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33686: BlockingRecv error: recv got EOF from 127.0.0.1:33686 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 00:58:49.352998 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 1620
I20260806 00:58:49.359687 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 1755
I20260806 00:58:49.366014 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 1887
I20260806 00:58:49.371474 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 1528
2026-08-06T00:58:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (728 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T00:58:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T00:58:49Z Disabled control of system clock
I20260806 00:58:49.405320 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:42999
--webserver_interface=127.23.242.62
--webserver_port=0
--builtin_ntp_servers=127.23.242.20:36055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:42999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 00:58:49.502561 2027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:49.502839 2027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:49.502866 2027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:49.504117 2027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 00:58:49.504163 2027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:49.504177 2027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:49.504191 2027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 00:58:49.504205 2027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 00:58:49.505705 2027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:36055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:42999
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:42999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:49.505928 2027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:49.506209 2027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:49.511801 2032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:49.511950 2035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:49.511833 2027 server_base.cc:1034] running on GCE node
W20260806 00:58:49.511833 2033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:49.512322 2027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:49.512671 2027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:49.513911 2027 hybrid_clock.cc:648] HybridClock initialized: now 1785977929513888 us; error 35 us; skew 500 ppm
I20260806 00:58:49.515707 2027 webserver.cc:460] Webserver started at http://127.23.242.62:46633/ using document root <none> and password file <none>
I20260806 00:58:49.516042 2027 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:49.516124 2027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:49.516290 2027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:49.517870 2027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "41f0634242bc40c7ace1857070d55034"
format_stamp: "Formatted at 2026-08-06 00:58:49 on dist-test-slave-dhph"
server_key: "39040fafd628bc834311ebef5d22d199"
server_key_iv: "7fc584b0d6fe7ff9ee7fc1a9e08f421b"
server_key_version: "encryptionkey@0"
I20260806 00:58:49.518296 2027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "41f0634242bc40c7ace1857070d55034"
format_stamp: "Formatted at 2026-08-06 00:58:49 on dist-test-slave-dhph"
server_key: "39040fafd628bc834311ebef5d22d199"
server_key_iv: "7fc584b0d6fe7ff9ee7fc1a9e08f421b"
server_key_version: "encryptionkey@0"
I20260806 00:58:49.520104 2027 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260806 00:58:49.521139 2041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:49.521461 2027 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:49.521580 2027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "41f0634242bc40c7ace1857070d55034"
format_stamp: "Formatted at 2026-08-06 00:58:49 on dist-test-slave-dhph"
server_key: "39040fafd628bc834311ebef5d22d199"
server_key_iv: "7fc584b0d6fe7ff9ee7fc1a9e08f421b"
server_key_version: "encryptionkey@0"
I20260806 00:58:49.521723 2027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:49.525718 2027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:49.525980 2027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:49.526081 2027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:49.530442 2027 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:42999
I20260806 00:58:49.530474 2093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:42999 every 8 connection(s)
I20260806 00:58:49.530982 2027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 00:58:49.531975 2094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:49.532505 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 2027
I20260806 00:58:49.532804 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 00:58:49.533059 24520 external_mini_cluster.cc:1428] Setting key 132e2585fc0296a9693bc1c57708fbb3
I20260806 00:58:49.536007 2097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:49.533732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57292 (local address 127.23.242.62:42999)
0806 00:58:49.533967 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:49.533978 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:49.534036 (+ 58us) server_negotiation.cc:407] Connection header received
0806 00:58:49.534507 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:49.534513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:49.534554 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:49.534760 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:49.535519 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:49.535530 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:49.535530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 00:58:49.535535 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:49.535603 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 00:58:49.535615 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:49.535693 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:49.535788 (+ 95us) server_negotiation.cc:299] Negotiation successful
0806 00:58:49.535850 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260806 00:58:49.536037 2094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f0634242bc40c7ace1857070d55034: Bootstrap starting.
I20260806 00:58:49.536948 2094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41f0634242bc40c7ace1857070d55034: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:49.537292 2094 log.cc:824] T 00000000000000000000000000000000 P 41f0634242bc40c7ace1857070d55034: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:49.538292 2094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f0634242bc40c7ace1857070d55034: No bootstrap required, opened a new log
I20260806 00:58:49.539868 2094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41f0634242bc40c7ace1857070d55034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f0634242bc40c7ace1857070d55034" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 42999 } }
I20260806 00:58:49.540076 2094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41f0634242bc40c7ace1857070d55034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:49.540120 2094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41f0634242bc40c7ace1857070d55034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41f0634242bc40c7ace1857070d55034, State: Initialized, Role: FOLLOWER
I20260806 00:58:49.540244 2094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41f0634242bc40c7ace1857070d55034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f0634242bc40c7ace1857070d55034" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 42999 } }
I20260806 00:58:49.540328 2094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41f0634242bc40c7ace1857070d55034 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 00:58:49.540371 2094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41f0634242bc40c7ace1857070d55034 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 00:58:49.540419 2094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41f0634242bc40c7ace1857070d55034 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:49.541353 2094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41f0634242bc40c7ace1857070d55034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f0634242bc40c7ace1857070d55034" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 42999 } }
I20260806 00:58:49.541481 2094 leader_election.cc:302] T 00000000000000000000000000000000 P 41f0634242bc40c7ace1857070d55034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41f0634242bc40c7ace1857070d55034; no voters:
I20260806 00:58:49.541687 2094 leader_election.cc:288] T 00000000000000000000000000000000 P 41f0634242bc40c7ace1857070d55034 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 00:58:49.541913 2099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41f0634242bc40c7ace1857070d55034 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:49.541972 2094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41f0634242bc40c7ace1857070d55034 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 00:58:49.542104 2099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41f0634242bc40c7ace1857070d55034 [term 1 LEADER]: Becoming Leader. State: Replica: 41f0634242bc40c7ace1857070d55034, State: Running, Role: LEADER
I20260806 00:58:49.542292 2099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41f0634242bc40c7ace1857070d55034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f0634242bc40c7ace1857070d55034" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 42999 } }
I20260806 00:58:49.542835 2101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f0634242bc40c7ace1857070d55034 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41f0634242bc40c7ace1857070d55034. Latest consensus state: current_term: 1 leader_uuid: "41f0634242bc40c7ace1857070d55034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f0634242bc40c7ace1857070d55034" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 42999 } } }
I20260806 00:58:49.542946 2101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f0634242bc40c7ace1857070d55034 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:49.543021 2100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f0634242bc40c7ace1857070d55034 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41f0634242bc40c7ace1857070d55034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41f0634242bc40c7ace1857070d55034" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 42999 } } }
I20260806 00:58:49.543082 2100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f0634242bc40c7ace1857070d55034 [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:49.543313 2109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 00:58:49.543797 2109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 00:58:49.545787 2109 catalog_manager.cc:1347] Generated new cluster ID: 67a5bc3b6cc549e9b85b005ca9eb327a
I20260806 00:58:49.545874 2109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 00:58:49.575856 2109 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 00:58:49.576071 2109 catalog_manager.cc:1504] Loading token signing keys...
I20260806 00:58:49.592731 2109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41f0634242bc40c7ace1857070d55034: Generated new TSK 0
I20260806 00:58:49.592975 2109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 00:58:49.602823 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:0
--local_ip_for_outbound_sockets=127.23.242.1
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:42999
--builtin_ntp_servers=127.23.242.20:36055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 00:58:49.712909 2119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:49.713267 2119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:49.713295 2119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:49.714484 2119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:49.714536 2119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:49.714565 2119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 00:58:49.715981 2119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:36055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:42999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:49.716187 2119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:49.716446 2119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:49.722067 2124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:49.722149 2125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:49.722105 2127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:49.722854 2119 server_base.cc:1034] running on GCE node
I20260806 00:58:49.723148 2119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:49.723423 2119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:49.724677 2119 hybrid_clock.cc:648] HybridClock initialized: now 1785977929724667 us; error 54 us; skew 500 ppm
I20260806 00:58:49.726775 2119 webserver.cc:460] Webserver started at http://127.23.242.1:42963/ using document root <none> and password file <none>
I20260806 00:58:49.727074 2119 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:49.727159 2119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:49.727329 2119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:49.728686 2119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/data/instance:
uuid: "9e81965bd7ab43728efe2431fce3794f"
format_stamp: "Formatted at 2026-08-06 00:58:49 on dist-test-slave-dhph"
server_key: "edcef18db40db930546334fafa8c0193"
server_key_iv: "8c5768e435b98474df72a612baeeb312"
server_key_version: "encryptionkey@0"
I20260806 00:58:49.729128 2119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance:
uuid: "9e81965bd7ab43728efe2431fce3794f"
format_stamp: "Formatted at 2026-08-06 00:58:49 on dist-test-slave-dhph"
server_key: "edcef18db40db930546334fafa8c0193"
server_key_iv: "8c5768e435b98474df72a612baeeb312"
server_key_version: "encryptionkey@0"
I20260806 00:58:49.730935 2119 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260806 00:58:49.732038 2133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:49.732262 2119 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:49.732389 2119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "9e81965bd7ab43728efe2431fce3794f"
format_stamp: "Formatted at 2026-08-06 00:58:49 on dist-test-slave-dhph"
server_key: "edcef18db40db930546334fafa8c0193"
server_key_iv: "8c5768e435b98474df72a612baeeb312"
server_key_version: "encryptionkey@0"
I20260806 00:58:49.732479 2119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:49.736949 2119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:49.737252 2119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:49.737416 2119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:49.737641 2119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:49.737977 2119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:49.738030 2119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:49.738072 2119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:49.738099 2119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:49.744992 2119 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:34507
I20260806 00:58:49.745074 2246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:34507 every 8 connection(s)
I20260806 00:58:49.745427 2119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 00:58:49.748853 2097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:49.746338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:37835 (local address 127.23.242.62:42999)
0806 00:58:49.746406 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:49.746408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:49.746653 (+ 245us) server_negotiation.cc:407] Connection header received
0806 00:58:49.747204 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:49.747206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:49.747212 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:49.747286 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:49.748598 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:49.748599 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:49.748599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 00:58:49.748604 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:49.748620 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 00:58:49.748624 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:49.748691 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:49.748782 (+ 91us) server_negotiation.cc:299] Negotiation successful
0806 00:58:49.748796 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 00:58:49.749073 2249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:49.746244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:42999 (local address 127.23.242.1:37835)
0806 00:58:49.746465 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:49.746508 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:49.747071 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:49.747358 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:49.747361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:49.747369 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:49.748430 (+ 1061us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 00:58:49.748431 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:49.748459 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 00:58:49.748487 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:49.748704 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:49.748709 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:49.748712 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 00:58:49.748773 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 00:58:49.748823 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":126,"threads_started":1}
I20260806 00:58:49.750027 2247 heartbeater.cc:347] Connected to a master server at 127.23.242.62:42999
I20260806 00:58:49.750204 2247 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:49.750560 2247 heartbeater.cc:511] Master 127.23.242.62:42999 requested a full tablet report, sending...
I20260806 00:58:49.750731 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 2119
I20260806 00:58:49.750874 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance
I20260806 00:58:49.751138 24520 external_mini_cluster.cc:1428] Setting key c7e4dba79e27931a7e491ed0d0a62bb9
I20260806 00:58:49.751525 2058 ts_manager.cc:194] Registered new tserver with Master: 9e81965bd7ab43728efe2431fce3794f (127.23.242.1:34507)
I20260806 00:58:49.752616 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:0
--local_ip_for_outbound_sockets=127.23.242.2
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:42999
--builtin_ntp_servers=127.23.242.20:36055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 00:58:49.842307 2250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:49.842559 2250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:49.842614 2250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:49.843950 2250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:49.843992 2250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:49.844022 2250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 00:58:49.845386 2250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:36055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:42999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:49.845649 2250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:49.845976 2250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:49.852384 2256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:49.852481 2259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:49.852643 2257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:49.852816 2250 server_base.cc:1034] running on GCE node
I20260806 00:58:49.853147 2250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:49.853530 2250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:49.854802 2250 hybrid_clock.cc:648] HybridClock initialized: now 1785977929854763 us; error 35 us; skew 500 ppm
I20260806 00:58:49.856412 2250 webserver.cc:460] Webserver started at http://127.23.242.2:35887/ using document root <none> and password file <none>
I20260806 00:58:49.856755 2250 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:49.856832 2250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:49.856987 2250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:49.858237 2250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/data/instance:
uuid: "823b0807370a43d4a1808226b96c966a"
format_stamp: "Formatted at 2026-08-06 00:58:49 on dist-test-slave-dhph"
server_key: "5d4de1ab791046189413b19c2bd1af4b"
server_key_iv: "7ef093b399c3b7aa74c4a4413d6bb688"
server_key_version: "encryptionkey@0"
I20260806 00:58:49.858685 2250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance:
uuid: "823b0807370a43d4a1808226b96c966a"
format_stamp: "Formatted at 2026-08-06 00:58:49 on dist-test-slave-dhph"
server_key: "5d4de1ab791046189413b19c2bd1af4b"
server_key_iv: "7ef093b399c3b7aa74c4a4413d6bb688"
server_key_version: "encryptionkey@0"
I20260806 00:58:49.860110 2250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:49.861105 2265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:49.861447 2250 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:49.861577 2250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "823b0807370a43d4a1808226b96c966a"
format_stamp: "Formatted at 2026-08-06 00:58:49 on dist-test-slave-dhph"
server_key: "5d4de1ab791046189413b19c2bd1af4b"
server_key_iv: "7ef093b399c3b7aa74c4a4413d6bb688"
server_key_version: "encryptionkey@0"
I20260806 00:58:49.861675 2250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:49.865397 2250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:49.865644 2250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:49.865756 2250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:49.865917 2250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:49.866247 2250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:49.866297 2250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:49.866319 2250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:49.866333 2250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:49.873494 2250 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:41033
I20260806 00:58:49.873600 2379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:41033 every 8 connection(s)
I20260806 00:58:49.873930 2250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 00:58:49.877015 2097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:49.874781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:52835 (local address 127.23.242.62:42999)
0806 00:58:49.874831 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:49.874833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:49.875091 (+ 258us) server_negotiation.cc:407] Connection header received
0806 00:58:49.875590 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:49.875591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:49.875597 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:49.875657 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:49.876774 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:49.876775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:49.876776 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 00:58:49.876780 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:49.876794 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 00:58:49.876798 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:49.876853 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:49.876941 (+ 88us) server_negotiation.cc:299] Negotiation successful
0806 00:58:49.876956 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 00:58:49.877172 2382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:49.874781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:42999 (local address 127.23.242.2:52835)
0806 00:58:49.874967 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:49.875001 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:49.875464 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:49.875707 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:49.875711 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:49.875718 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:49.876624 (+ 906us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 00:58:49.876624 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:49.876652 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 00:58:49.876679 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:49.876854 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:49.876856 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:49.876859 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 00:58:49.876911 (+ 52us) client_negotiation.cc:240] Negotiation successful
0806 00:58:49.876958 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
I20260806 00:58:49.877732 2380 heartbeater.cc:347] Connected to a master server at 127.23.242.62:42999
I20260806 00:58:49.877825 2380 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:49.877997 2380 heartbeater.cc:511] Master 127.23.242.62:42999 requested a full tablet report, sending...
I20260806 00:58:49.878789 2057 ts_manager.cc:194] Registered new tserver with Master: 823b0807370a43d4a1808226b96c966a (127.23.242.2:41033)
I20260806 00:58:49.879084 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 2250
I20260806 00:58:49.879257 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance
I20260806 00:58:49.879489 24520 external_mini_cluster.cc:1428] Setting key 7767cb81533a6c32be399bb601fb8561
I20260806 00:58:49.880821 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:0
--local_ip_for_outbound_sockets=127.23.242.3
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:42999
--builtin_ntp_servers=127.23.242.20:36055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 00:58:49.977291 2383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:49.977524 2383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:49.977550 2383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:49.978845 2383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:49.978899 2383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:49.978926 2383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 00:58:49.980404 2383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:36055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:42999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:49.980628 2383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:49.980891 2383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:49.986645 2392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:49.986652 2389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:49.986770 2390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:49.987133 2383 server_base.cc:1034] running on GCE node
I20260806 00:58:49.987344 2383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:49.987598 2383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:49.988813 2383 hybrid_clock.cc:648] HybridClock initialized: now 1785977929988777 us; error 66 us; skew 500 ppm
I20260806 00:58:49.990907 2383 webserver.cc:460] Webserver started at http://127.23.242.3:41253/ using document root <none> and password file <none>
I20260806 00:58:49.991252 2383 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:49.991340 2383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:49.991537 2383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:49.993204 2383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/data/instance:
uuid: "2dee230e038248039f07ff51e82b2ebc"
format_stamp: "Formatted at 2026-08-06 00:58:49 on dist-test-slave-dhph"
server_key: "659ce9d6ae2105af5ecee635aae45469"
server_key_iv: "2ed0716cbf2270ccf4636ca1bab141c5"
server_key_version: "encryptionkey@0"
I20260806 00:58:49.993758 2383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance:
uuid: "2dee230e038248039f07ff51e82b2ebc"
format_stamp: "Formatted at 2026-08-06 00:58:49 on dist-test-slave-dhph"
server_key: "659ce9d6ae2105af5ecee635aae45469"
server_key_iv: "2ed0716cbf2270ccf4636ca1bab141c5"
server_key_version: "encryptionkey@0"
I20260806 00:58:49.995872 2383 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260806 00:58:49.997117 2398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:49.997375 2383 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:49.997484 2383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "2dee230e038248039f07ff51e82b2ebc"
format_stamp: "Formatted at 2026-08-06 00:58:49 on dist-test-slave-dhph"
server_key: "659ce9d6ae2105af5ecee635aae45469"
server_key_iv: "2ed0716cbf2270ccf4636ca1bab141c5"
server_key_version: "encryptionkey@0"
I20260806 00:58:49.997546 2383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:50.001338 2383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:50.001732 2383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:50.001904 2383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:50.002147 2383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:50.002501 2383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:50.002564 2383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:50.002632 2383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:50.002681 2383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:50.010332 2383 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:41195
I20260806 00:58:50.010370 2514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:41195 every 8 connection(s)
I20260806 00:58:50.010912 2383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
I20260806 00:58:50.014654 2517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:50.011730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:42999 (local address 127.23.242.3:48671)
0806 00:58:50.011995 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:50.012029 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:50.012485 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:50.012949 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:50.012952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:50.012960 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:50.013966 (+ 1006us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 00:58:50.013967 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:50.013997 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 00:58:50.014025 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:50.014255 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:50.014262 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:50.014264 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 00:58:50.014354 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 00:58:50.014412 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":94,"threads_started":1}
I20260806 00:58:50.014777 2097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:50.011825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:48671 (local address 127.23.242.62:42999)
0806 00:58:50.011895 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:50.011898 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:50.012138 (+ 240us) server_negotiation.cc:407] Connection header received
0806 00:58:50.012748 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:50.012749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:50.012787 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:50.012854 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:50.014131 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:50.014132 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:50.014132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 00:58:50.014136 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:50.014151 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 00:58:50.014156 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:50.014230 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:50.014436 (+ 206us) server_negotiation.cc:299] Negotiation successful
0806 00:58:50.014471 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 00:58:50.015255 2515 heartbeater.cc:347] Connected to a master server at 127.23.242.62:42999
I20260806 00:58:50.015355 2515 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:50.015482 2515 heartbeater.cc:511] Master 127.23.242.62:42999 requested a full tablet report, sending...
I20260806 00:58:50.015995 2057 ts_manager.cc:194] Registered new tserver with Master: 2dee230e038248039f07ff51e82b2ebc (127.23.242.3:41195)
I20260806 00:58:50.017194 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 2383
I20260806 00:58:50.017305 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance
I20260806 00:58:50.017565 24520 external_mini_cluster.cc:1428] Setting key 4fb6c3fc840b2f8574e4cc1f80ce7e43
I20260806 00:58:50.019202 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 00:58:50.021495 2097 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 00:58:50.020752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57306 (local address 127.23.242.62:42999)
0806 00:58:50.020942 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:50.020944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:50.021029 (+ 85us) server_negotiation.cc:407] Connection header received
0806 00:58:50.021130 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:50.021131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:50.021137 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:50.021209 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:50.021418 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57306: BlockingRecv error: recv got EOF from 127.0.0.1:57306 (error 108)
Metrics: {"server-negotiator.queue_time_us":158}
I20260806 00:58:50.021641 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 2119
I20260806 00:58:50.030483 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 2250
I20260806 00:58:50.037639 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 2383
I20260806 00:58:50.044327 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 2027
2026-08-06T00:58:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (675 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 00:58:50 dist-test-slave-dhph krb5kdc[2525](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 00:58:50 dist-test-slave-dhph krb5kdc[2525](info): set up 2 sockets
Aug 06 00:58:50 dist-test-slave-dhph krb5kdc[2525](info): commencing operation
krb5kdc: starting...
W20260806 00:58:51.174898 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 00:58:51 dist-test-slave-dhph krb5kdc[2525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T00:58:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T00:58:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 00:58:51.315268 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:38759
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:38379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:38759
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:51.410116 2538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:51.410429 2538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:51.410475 2538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:51.411902 2538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 00:58:51.411998 2538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:51.412026 2538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:51.412050 2538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 00:58:51.412084 2538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 00:58:51.414302 2538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:38379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:38759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:38759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:51.414654 2538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:51.414918 2538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:51.420266 2546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:51.420444 2538 server_base.cc:1034] running on GCE node
W20260806 00:58:51.420531 2543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:51.420794 2544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:51.426976 2538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:51.427405 2538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:51.428665 2538 hybrid_clock.cc:648] HybridClock initialized: now 1785977931428600 us; error 89 us; skew 500 ppm
Aug 06 00:58:51 dist-test-slave-dhph krb5kdc[2525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977931, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:51.432420 2538 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 00:58:51.432988 2538 webserver.cc:460] Webserver started at http://127.23.242.62:37773/ using document root <none> and password file <none>
I20260806 00:58:51.433262 2538 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:51.433342 2538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:51.433508 2538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:51.434898 2538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "74f2202f01f84b03af55b671b2afb10d"
format_stamp: "Formatted at 2026-08-06 00:58:51 on dist-test-slave-dhph"
server_key: "5dc3cbf734fe955b5acf28ca0c380bfc"
server_key_iv: "7bda74b0d3330951a06edb4b863bad2a"
server_key_version: "encryptionkey@0"
I20260806 00:58:51.435307 2538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "74f2202f01f84b03af55b671b2afb10d"
format_stamp: "Formatted at 2026-08-06 00:58:51 on dist-test-slave-dhph"
server_key: "5dc3cbf734fe955b5acf28ca0c380bfc"
server_key_iv: "7bda74b0d3330951a06edb4b863bad2a"
server_key_version: "encryptionkey@0"
I20260806 00:58:51.436961 2538 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 00:58:51.438051 2553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:51.438306 2538 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:51.438436 2538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "74f2202f01f84b03af55b671b2afb10d"
format_stamp: "Formatted at 2026-08-06 00:58:51 on dist-test-slave-dhph"
server_key: "5dc3cbf734fe955b5acf28ca0c380bfc"
server_key_iv: "7bda74b0d3330951a06edb4b863bad2a"
server_key_version: "encryptionkey@0"
I20260806 00:58:51.438524 2538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:51.449589 2538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:51.452476 2538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:51.452708 2538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:51.458647 2538 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:38759
I20260806 00:58:51.458699 2606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:38759 every 8 connection(s)
I20260806 00:58:51.459082 2538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 00:58:51.460229 2607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:51.462113 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 2538
I20260806 00:58:51.462240 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 00:58:51.462458 24520 external_mini_cluster.cc:1428] Setting key 77e9e1dd1ed4bf7170e502e0261221d6
I20260806 00:58:51.463322 2607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f2202f01f84b03af55b671b2afb10d: Bootstrap starting.
I20260806 00:58:51.464063 2607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74f2202f01f84b03af55b671b2afb10d: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:51.464416 2607 log.cc:824] T 00000000000000000000000000000000 P 74f2202f01f84b03af55b671b2afb10d: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:51.465312 2607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f2202f01f84b03af55b671b2afb10d: No bootstrap required, opened a new log
I20260806 00:58:51.467823 2607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74f2202f01f84b03af55b671b2afb10d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f2202f01f84b03af55b671b2afb10d" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38759 } }
I20260806 00:58:51.468027 2607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74f2202f01f84b03af55b671b2afb10d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:51.468076 2607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74f2202f01f84b03af55b671b2afb10d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f2202f01f84b03af55b671b2afb10d, State: Initialized, Role: FOLLOWER
I20260806 00:58:51.468274 2607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74f2202f01f84b03af55b671b2afb10d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f2202f01f84b03af55b671b2afb10d" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38759 } }
I20260806 00:58:51.468410 2607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74f2202f01f84b03af55b671b2afb10d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 00:58:51.468462 2607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74f2202f01f84b03af55b671b2afb10d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 00:58:51.468535 2607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74f2202f01f84b03af55b671b2afb10d [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:51.469187 2607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74f2202f01f84b03af55b671b2afb10d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f2202f01f84b03af55b671b2afb10d" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38759 } }
I20260806 00:58:51.469319 2607 leader_election.cc:302] T 00000000000000000000000000000000 P 74f2202f01f84b03af55b671b2afb10d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74f2202f01f84b03af55b671b2afb10d; no voters:
I20260806 00:58:51.469525 2607 leader_election.cc:288] T 00000000000000000000000000000000 P 74f2202f01f84b03af55b671b2afb10d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 00:58:51.469580 2612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74f2202f01f84b03af55b671b2afb10d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:51.469738 2612 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74f2202f01f84b03af55b671b2afb10d [term 1 LEADER]: Becoming Leader. State: Replica: 74f2202f01f84b03af55b671b2afb10d, State: Running, Role: LEADER
I20260806 00:58:51.469818 2607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74f2202f01f84b03af55b671b2afb10d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 00:58:51.469851 2612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74f2202f01f84b03af55b671b2afb10d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f2202f01f84b03af55b671b2afb10d" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38759 } }
I20260806 00:58:51.470196 2613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f2202f01f84b03af55b671b2afb10d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74f2202f01f84b03af55b671b2afb10d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f2202f01f84b03af55b671b2afb10d" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38759 } } }
I20260806 00:58:51.470359 2613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f2202f01f84b03af55b671b2afb10d [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:51.470184 2612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f2202f01f84b03af55b671b2afb10d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74f2202f01f84b03af55b671b2afb10d. Latest consensus state: current_term: 1 leader_uuid: "74f2202f01f84b03af55b671b2afb10d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f2202f01f84b03af55b671b2afb10d" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38759 } } }
Aug 06 00:58:51 dist-test-slave-dhph krb5kdc[2525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:51.470558 2612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f2202f01f84b03af55b671b2afb10d [sys.catalog]: This master's current role is: LEADER
I20260806 00:58:51.470722 2619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 00:58:51.471264 2619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 00:58:51.473635 2619 catalog_manager.cc:1347] Generated new cluster ID: 6c9b4eba30654198842eba205fb39742
I20260806 00:58:51.473691 2619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 00:58:51.475689 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:51.463118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37296 (local address 127.23.242.62:38759)
0806 00:58:51.463340 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:51.463348 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:51.463395 (+ 47us) server_negotiation.cc:407] Connection header received
0806 00:58:51.463885 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:51.463892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:51.463940 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:51.464101 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:51.464840 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:51.466312 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:51.467292 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:51.467955 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:51.468099 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:51.471232 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:51.471236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:51.471237 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:51.471250 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:51.473950 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:51.474383 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:51.474386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:51.474387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:51.474495 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:51.474751 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:51.474752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:51.474752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:51.475042 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:51.475178 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:51.475320 (+ 142us) server_negotiation.cc:299] Negotiation successful
0806 00:58:51.475456 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":101,"threads_started":1}
I20260806 00:58:51.487562 2619 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 00:58:51.488363 2619 catalog_manager.cc:1504] Loading token signing keys...
I20260806 00:58:51.496013 2619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74f2202f01f84b03af55b671b2afb10d: Generated new TSK 0
I20260806 00:58:51.496295 2619 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.242.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
I20260806 00:58:51.535869 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:0
--local_ip_for_outbound_sockets=127.23.242.1
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:38759
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:38379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:51.636585 2633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:51.636845 2633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:51.636876 2633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:51.638276 2633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:51.638337 2633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:51.638370 2633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 00:58:51.640110 2633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:38379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:38759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:51.640592 2633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:51.641085 2633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:51.647851 2638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:51.647935 2641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:51.648047 2639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:51.648324 2633 server_base.cc:1034] running on GCE node
I20260806 00:58:51.648834 2633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:51.649055 2633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:51.650460 2633 hybrid_clock.cc:648] HybridClock initialized: now 1785977931650424 us; error 67 us; skew 500 ppm
Aug 06 00:58:51 dist-test-slave-dhph krb5kdc[2525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977931, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:51.653469 2633 init.cc:377] Logged in from keytab as kudu/127.23.242.1@KRBTEST.COM (short username kudu)
I20260806 00:58:51.653990 2633 webserver.cc:460] Webserver started at http://127.23.242.1:46691/ using document root <none> and password file <none>
I20260806 00:58:51.654238 2633 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:51.654320 2633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:51.654477 2633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:51.655668 2633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/data/instance:
uuid: "5b79d6af945d4929922e97b4d9725de0"
format_stamp: "Formatted at 2026-08-06 00:58:51 on dist-test-slave-dhph"
server_key: "f9a70c0c94b6f7547c6340acd840a167"
server_key_iv: "f40cf1875a411c5c60ba6fef05002ebd"
server_key_version: "encryptionkey@0"
I20260806 00:58:51.656055 2633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance:
uuid: "5b79d6af945d4929922e97b4d9725de0"
format_stamp: "Formatted at 2026-08-06 00:58:51 on dist-test-slave-dhph"
server_key: "f9a70c0c94b6f7547c6340acd840a167"
server_key_iv: "f40cf1875a411c5c60ba6fef05002ebd"
server_key_version: "encryptionkey@0"
I20260806 00:58:51.657549 2633 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 00:58:51.658557 2649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:51.658929 2633 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:51.659045 2633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "5b79d6af945d4929922e97b4d9725de0"
format_stamp: "Formatted at 2026-08-06 00:58:51 on dist-test-slave-dhph"
server_key: "f9a70c0c94b6f7547c6340acd840a167"
server_key_iv: "f40cf1875a411c5c60ba6fef05002ebd"
server_key_version: "encryptionkey@0"
I20260806 00:58:51.659129 2633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:51.690804 2633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:51.693534 2633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:51.693665 2633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:51.693888 2633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:51.694238 2633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:51.694278 2633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:51.694303 2633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:51.694314 2633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:51.701278 2633 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:46833
I20260806 00:58:51.701362 2762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:46833 every 8 connection(s)
I20260806 00:58:51.701723 2633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 00:58:51.703944 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 2633
I20260806 00:58:51.704095 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance
I20260806 00:58:51.704382 24520 external_mini_cluster.cc:1428] Setting key d38d2626be9cdd7e56496a86f26a8b4d
Aug 06 00:58:51 dist-test-slave-dhph krb5kdc[2525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977931, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:51.713335 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:51.702686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:53387 (local address 127.23.242.62:38759)
0806 00:58:51.702781 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:51.702783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:51.702996 (+ 213us) server_negotiation.cc:407] Connection header received
0806 00:58:51.703651 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:51.703652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:51.703666 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:51.703748 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:51.705220 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:51.705830 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:51.707095 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:51.707330 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:51.707409 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:51.709980 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:51.709982 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:51.709982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:51.709997 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:51.711638 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:51.712235 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:51.712236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:51.712237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:51.712302 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:51.712587 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:51.712588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:51.712589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:51.712724 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:51.712809 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:51.713124 (+ 315us) server_negotiation.cc:299] Negotiation successful
0806 00:58:51.713232 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 00:58:51.713572 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:51.702547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38759 (local address 127.23.242.1:53387)
0806 00:58:51.702858 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:51.702898 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:51.703443 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:51.703892 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:51.703898 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:51.704792 (+ 894us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:51.705093 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:51.705100 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:51.705944 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:51.705945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:51.706993 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:51.706996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:51.707434 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:51.707435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:51.707571 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:51.708319 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:51.708328 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:51.709853 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:51.711748 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:51.711757 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:51.711760 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:51.712134 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:51.712379 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:51.712379 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:51.712380 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:51.712508 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:51.712799 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:51.712807 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:51.712969 (+ 162us) client_negotiation.cc:769] Sending connection context
0806 00:58:51.713093 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 00:58:51.713275 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":153,"threads_started":1}
WARNING: no policy specified for kudu/127.23.242.2@KRBTEST.COM; defaulting to no policy
I20260806 00:58:51.714190 2763 heartbeater.cc:347] Connected to a master server at 127.23.242.62:38759
I20260806 00:58:51.714565 2763 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:51.715119 2763 heartbeater.cc:511] Master 127.23.242.62:38759 requested a full tablet report, sending...
I20260806 00:58:51.715966 2570 ts_manager.cc:194] Registered new tserver with Master: 5b79d6af945d4929922e97b4d9725de0 (127.23.242.1:46833)
I20260806 00:58:51.716675 2570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.1@KRBTEST.COM'} at 127.23.242.1:53387
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
I20260806 00:58:51.739600 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:0
--local_ip_for_outbound_sockets=127.23.242.2
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:38759
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:38379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:51.832438 2768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:51.832674 2768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:51.832700 2768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:51.834406 2768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:51.834472 2768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:51.834503 2768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 00:58:51.836215 2768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:38379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:38759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:51.836489 2768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:51.836764 2768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:51.842005 2776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:51.842061 2774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:51.842433 2768 server_base.cc:1034] running on GCE node
W20260806 00:58:51.842223 2773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:51.842799 2768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:51.843143 2768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:51.844350 2768 hybrid_clock.cc:648] HybridClock initialized: now 1785977931844336 us; error 30 us; skew 500 ppm
Aug 06 00:58:51 dist-test-slave-dhph krb5kdc[2525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977931, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:51.847620 2768 init.cc:377] Logged in from keytab as kudu/127.23.242.2@KRBTEST.COM (short username kudu)
I20260806 00:58:51.848237 2768 webserver.cc:460] Webserver started at http://127.23.242.2:41571/ using document root <none> and password file <none>
I20260806 00:58:51.848501 2768 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:51.848583 2768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:51.848757 2768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:51.850060 2768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/data/instance:
uuid: "163d0412a9904986999c9c19bffb4aa1"
format_stamp: "Formatted at 2026-08-06 00:58:51 on dist-test-slave-dhph"
server_key: "1f5fb7c9ab9dd724c3b128dc4131f4a4"
server_key_iv: "5b391173df63ec372a5a7a433aca316c"
server_key_version: "encryptionkey@0"
I20260806 00:58:51.850525 2768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance:
uuid: "163d0412a9904986999c9c19bffb4aa1"
format_stamp: "Formatted at 2026-08-06 00:58:51 on dist-test-slave-dhph"
server_key: "1f5fb7c9ab9dd724c3b128dc4131f4a4"
server_key_iv: "5b391173df63ec372a5a7a433aca316c"
server_key_version: "encryptionkey@0"
I20260806 00:58:51.852186 2768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:51.853174 2783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:51.853477 2768 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:58:51.853636 2768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "163d0412a9904986999c9c19bffb4aa1"
format_stamp: "Formatted at 2026-08-06 00:58:51 on dist-test-slave-dhph"
server_key: "1f5fb7c9ab9dd724c3b128dc4131f4a4"
server_key_iv: "5b391173df63ec372a5a7a433aca316c"
server_key_version: "encryptionkey@0"
I20260806 00:58:51.853726 2768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:51.877627 2768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:51.880693 2768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:51.880838 2768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:51.881136 2768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:51.881526 2768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:51.881564 2768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:51.881610 2768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:51.881641 2768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:51.889278 2768 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:38843
I20260806 00:58:51.889343 2897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:38843 every 8 connection(s)
I20260806 00:58:51.889696 2768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 00:58:51.896828 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 2768
I20260806 00:58:51.896986 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance
Aug 06 00:58:51 dist-test-slave-dhph krb5kdc[2525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977931, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:51.897468 24520 external_mini_cluster.cc:1428] Setting key 35759de381b7fd0ee99b02f66b1bde8e
I20260806 00:58:51.900871 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:51.890720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:43671 (local address 127.23.242.62:38759)
0806 00:58:51.890805 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:51.890808 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:51.890869 (+ 61us) server_negotiation.cc:407] Connection header received
0806 00:58:51.891450 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:51.891452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:51.891467 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:51.891552 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:51.892741 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:51.893269 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:51.894431 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:51.894740 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:51.894806 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:51.897785 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:51.897787 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:51.897788 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:51.897806 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:51.899353 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:51.899821 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:51.899822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:51.899822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:51.899891 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:51.900162 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:51.900163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:51.900163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:51.900285 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:51.900362 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:51.900693 (+ 331us) server_negotiation.cc:299] Negotiation successful
0806 00:58:51.900779 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 00:58:51.901136 2900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:51.890528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38759 (local address 127.23.242.2:43671)
0806 00:58:51.890733 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:51.890780 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:51.891315 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:51.891709 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:51.891713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:51.892375 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:51.892603 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:51.892608 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:51.893434 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:51.893435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:51.894315 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:51.894317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:51.894823 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:51.894824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:51.894976 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:51.895717 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:51.895727 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:51.897663 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:51.899471 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:51.899479 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:51.899482 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:51.899737 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:51.899970 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:51.899971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:51.899971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:51.900085 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:51.900354 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:51.900361 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:51.900514 (+ 153us) client_negotiation.cc:769] Sending connection context
0806 00:58:51.900655 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 00:58:51.900824 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":129,"threads_started":1}
I20260806 00:58:51.901785 2898 heartbeater.cc:347] Connected to a master server at 127.23.242.62:38759
I20260806 00:58:51.901922 2898 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:51.902130 2898 heartbeater.cc:511] Master 127.23.242.62:38759 requested a full tablet report, sending...
I20260806 00:58:51.902633 2570 ts_manager.cc:194] Registered new tserver with Master: 163d0412a9904986999c9c19bffb4aa1 (127.23.242.2:38843)
I20260806 00:58:51.903259 2570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.2@KRBTEST.COM'} at 127.23.242.2:43671
WARNING: no policy specified for kudu/127.23.242.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
I20260806 00:58:51.931950 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:0
--local_ip_for_outbound_sockets=127.23.242.3
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:38759
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:38379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:58:52.032946 2903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:58:52.033203 2903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:58:52.033236 2903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:58:52.034641 2903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:58:52.034698 2903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:58:52.034727 2903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 00:58:52.036289 2903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:38379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:38759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:58:52.036571 2903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:58:52.036823 2903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:58:52.042927 2908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:52.043042 2911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:58:52.043138 2909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:58:52.044251 2903 server_base.cc:1034] running on GCE node
I20260806 00:58:52.044526 2903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:58:52.044829 2903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:58:52.046005 2903 hybrid_clock.cc:648] HybridClock initialized: now 1785977932045993 us; error 37 us; skew 500 ppm
Aug 06 00:58:52 dist-test-slave-dhph krb5kdc[2525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977932, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:58:52.049515 2903 init.cc:377] Logged in from keytab as kudu/127.23.242.3@KRBTEST.COM (short username kudu)
I20260806 00:58:52.050386 2903 webserver.cc:460] Webserver started at http://127.23.242.3:35991/ using document root <none> and password file <none>
I20260806 00:58:52.050792 2903 fs_manager.cc:359] Metadata directory not provided
I20260806 00:58:52.050858 2903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:58:52.050972 2903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 00:58:52.052173 2903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/data/instance:
uuid: "a57bde86740e4fdaaf06588405ec2549"
format_stamp: "Formatted at 2026-08-06 00:58:52 on dist-test-slave-dhph"
server_key: "7070a13816a67650f4cc9ba1f03bea87"
server_key_iv: "38849f395494b80d0c949817c3383157"
server_key_version: "encryptionkey@0"
I20260806 00:58:52.052531 2903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance:
uuid: "a57bde86740e4fdaaf06588405ec2549"
format_stamp: "Formatted at 2026-08-06 00:58:52 on dist-test-slave-dhph"
server_key: "7070a13816a67650f4cc9ba1f03bea87"
server_key_iv: "38849f395494b80d0c949817c3383157"
server_key_version: "encryptionkey@0"
I20260806 00:58:52.054201 2903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260806 00:58:52.055603 2918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:52.055915 2903 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 00:58:52.056049 2903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "a57bde86740e4fdaaf06588405ec2549"
format_stamp: "Formatted at 2026-08-06 00:58:52 on dist-test-slave-dhph"
server_key: "7070a13816a67650f4cc9ba1f03bea87"
server_key_iv: "38849f395494b80d0c949817c3383157"
server_key_version: "encryptionkey@0"
I20260806 00:58:52.056138 2903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:58:52.070837 2903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:58:52.074015 2903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:58:52.074219 2903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:58:52.074501 2903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:58:52.074884 2903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:58:52.074944 2903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:52.074982 2903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:58:52.075031 2903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:58:52.081157 2903 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:35465
I20260806 00:58:52.081212 3032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:35465 every 8 connection(s)
I20260806 00:58:52.081537 2903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
Aug 06 00:58:52 dist-test-slave-dhph krb5kdc[2525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977932, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:58:52.089277 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 2903
I20260806 00:58:52.089393 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance
I20260806 00:58:52.089596 24520 external_mini_cluster.cc:1428] Setting key 5a5a8b123c8c5c7adee6b18bda11c0ad
I20260806 00:58:52.093248 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:52.082567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:54597 (local address 127.23.242.62:38759)
0806 00:58:52.082672 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:52.082675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:52.082948 (+ 273us) server_negotiation.cc:407] Connection header received
0806 00:58:52.083449 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:52.083450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:52.083462 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:52.083531 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:52.084512 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.085113 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.086242 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.086476 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.086548 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:52.089326 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:52.089328 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:52.089328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:52.089338 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:52.091702 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:52.092214 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:52.092216 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:52.092216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:52.092286 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:52.092551 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:52.092552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:52.092552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:52.092693 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:52.092771 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:52.093054 (+ 283us) server_negotiation.cc:299] Negotiation successful
0806 00:58:52.093146 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 00:58:52.093483 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:52.082555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:38759 (local address 127.23.242.3:54597)
0806 00:58:52.082823 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:52.082861 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:52.083337 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:52.083600 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:52.083603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:52.084213 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:58:52.084412 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:52.084416 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.085240 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.085242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:52.086137 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:52.086139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.086674 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.086675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:52.086807 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:52.087458 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:58:52.087468 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:58:52.089222 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:58:52.091795 (+ 2573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:52.091806 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:52.091808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:52.092115 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:52.092353 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:58:52.092354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:58:52.092354 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:58:52.092470 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:58:52.092762 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:58:52.092770 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:58:52.092915 (+ 145us) client_negotiation.cc:769] Sending connection context
0806 00:58:52.093032 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 00:58:52.093215 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":153,"threads_started":1}
I20260806 00:58:52.094017 3033 heartbeater.cc:347] Connected to a master server at 127.23.242.62:38759
I20260806 00:58:52.094127 3033 heartbeater.cc:464] Registering TS with master...
I20260806 00:58:52.094297 3033 heartbeater.cc:511] Master 127.23.242.62:38759 requested a full tablet report, sending...
I20260806 00:58:52.094722 2570 ts_manager.cc:194] Registered new tserver with Master: a57bde86740e4fdaaf06588405ec2549 (127.23.242.3:35465)
I20260806 00:58:52.095124 2570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.3@KRBTEST.COM'} at 127.23.242.3:54597
I20260806 00:58:52.103562 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 00:58:52.113246 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:52.105344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.23.242.62:38759)
0806 00:58:52.105470 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:52.105473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:52.105588 (+ 115us) server_negotiation.cc:407] Connection header received
0806 00:58:52.105709 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:52.105710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:52.105724 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:52.105791 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:58:52.106460 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.107090 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.107915 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.108127 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.108263 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:52.110055 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:58:52.110057 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:58:52.110057 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:58:52.110067 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:58:52.111765 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:52.112262 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:52.112263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:52.112264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:52.112333 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:58:52.112562 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:58:52.112563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:58:52.112563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:58:52.112750 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:58:52.112873 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:52.113014 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 00:58:52.113110 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 00:58:52.115824 2570 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 00:58:52.117607 2570 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 00:58:52.126194 3049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:52.121320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55732 (local address 127.23.242.3:35465)
0806 00:58:52.121524 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:52.121528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:52.121576 (+ 48us) server_negotiation.cc:407] Connection header received
0806 00:58:52.121654 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:52.121657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:52.121675 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:52.121780 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:52.121991 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.123319 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.124889 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.125671 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.125765 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:52.125832 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:52.126030 (+ 198us) server_negotiation.cc:299] Negotiation successful
0806 00:58:52.126081 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":108,"threads_started":1}
I20260806 00:58:52.126214 3044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:52.121016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:35465 (local address 127.0.0.1:55732)
0806 00:58:52.121510 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:52.121522 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:52.121580 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:52.121788 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:52.121789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:52.121791 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:52.121837 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:52.121840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.123442 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.123444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:52.124810 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:52.124812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.125801 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.125802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:52.125878 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:52.125891 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 00:58:52.126009 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 00:58:52.126073 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":5,"thread_start_us":123,"threads_started":1}
I20260806 00:58:52.127296 3045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:52.121015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:46833 (local address 127.0.0.1:58624)
0806 00:58:52.121239 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:52.121286 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:52.121363 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:52.121662 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:52.121665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:52.121667 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:52.121920 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:52.121922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.123099 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.123101 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:52.124805 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:52.124808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.127015 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.127016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:52.127089 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:52.127093 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 00:58:52.127162 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 00:58:52.127219 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":61,"thread_start_us":83,"threads_started":1}
I20260806 00:58:52.127180 2967 tablet_service.cc:1467] Processing CreateTablet for tablet 29aab19c1faa4202ae3b43da5f7088c1 (DEFAULT_TABLE table=test-table [id=686bf5aa3c94449aa2b1b5220276f9e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:52.127403 3048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:52.121231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58624 (local address 127.23.242.1:46833)
0806 00:58:52.121465 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:52.121470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:52.121484 (+ 14us) server_negotiation.cc:407] Connection header received
0806 00:58:52.121534 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:52.121536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:52.121555 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:52.121658 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:52.122054 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.122985 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.125889 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.126936 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.127008 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:52.127047 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:52.127240 (+ 193us) server_negotiation.cc:299] Negotiation successful
0806 00:58:52.127284 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":109,"threads_started":1}
I20260806 00:58:52.127610 2967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29aab19c1faa4202ae3b43da5f7088c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:52.128088 2697 tablet_service.cc:1467] Processing CreateTablet for tablet 29aab19c1faa4202ae3b43da5f7088c1 (DEFAULT_TABLE table=test-table [id=686bf5aa3c94449aa2b1b5220276f9e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:52.128458 2697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29aab19c1faa4202ae3b43da5f7088c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:52.131035 3046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:52.121016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:38843 (local address 127.0.0.1:51874)
0806 00:58:52.121499 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:52.121509 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:52.121556 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:52.122085 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:52.122086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:52.122089 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:52.122135 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:52.122137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.123587 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.123588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:52.124733 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:52.124736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.130747 (+ 6011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.130749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:52.130812 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:52.130818 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 00:58:52.130902 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 00:58:52.130955 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":87,"thread_start_us":125,"threads_started":1}
I20260806 00:58:52.131069 3047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:52.121226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51874 (local address 127.23.242.2:38843)
0806 00:58:52.121805 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:52.121810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:52.121820 (+ 10us) server_negotiation.cc:407] Connection header received
0806 00:58:52.121860 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:52.121862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:52.121878 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:52.121984 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:52.122245 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.123458 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.124833 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.125671 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.125769 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:52.125832 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:52.130930 (+ 5098us) server_negotiation.cc:299] Negotiation successful
0806 00:58:52.130989 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":125,"threads_started":1}
I20260806 00:58:52.131881 3050 tablet_bootstrap.cc:492] T 29aab19c1faa4202ae3b43da5f7088c1 P a57bde86740e4fdaaf06588405ec2549: Bootstrap starting.
I20260806 00:58:52.132066 2832 tablet_service.cc:1467] Processing CreateTablet for tablet 29aab19c1faa4202ae3b43da5f7088c1 (DEFAULT_TABLE table=test-table [id=686bf5aa3c94449aa2b1b5220276f9e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:58:52.132428 2832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29aab19c1faa4202ae3b43da5f7088c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:58:52.132627 3050 tablet_bootstrap.cc:654] T 29aab19c1faa4202ae3b43da5f7088c1 P a57bde86740e4fdaaf06588405ec2549: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:52.132644 3051 tablet_bootstrap.cc:492] T 29aab19c1faa4202ae3b43da5f7088c1 P 5b79d6af945d4929922e97b4d9725de0: Bootstrap starting.
I20260806 00:58:52.133203 3050 log.cc:824] T 29aab19c1faa4202ae3b43da5f7088c1 P a57bde86740e4fdaaf06588405ec2549: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:52.133262 3051 tablet_bootstrap.cc:654] T 29aab19c1faa4202ae3b43da5f7088c1 P 5b79d6af945d4929922e97b4d9725de0: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:52.133875 3051 log.cc:824] T 29aab19c1faa4202ae3b43da5f7088c1 P 5b79d6af945d4929922e97b4d9725de0: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:52.134227 3050 tablet_bootstrap.cc:492] T 29aab19c1faa4202ae3b43da5f7088c1 P a57bde86740e4fdaaf06588405ec2549: No bootstrap required, opened a new log
I20260806 00:58:52.134311 3050 ts_tablet_manager.cc:1397] T 29aab19c1faa4202ae3b43da5f7088c1 P a57bde86740e4fdaaf06588405ec2549: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 00:58:52.134727 3051 tablet_bootstrap.cc:492] T 29aab19c1faa4202ae3b43da5f7088c1 P 5b79d6af945d4929922e97b4d9725de0: No bootstrap required, opened a new log
I20260806 00:58:52.134785 3051 ts_tablet_manager.cc:1397] T 29aab19c1faa4202ae3b43da5f7088c1 P 5b79d6af945d4929922e97b4d9725de0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 00:58:52.135749 3050 raft_consensus.cc:348] T 29aab19c1faa4202ae3b43da5f7088c1 P a57bde86740e4fdaaf06588405ec2549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "163d0412a9904986999c9c19bffb4aa1" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38843 } } peers { permanent_uuid: "5b79d6af945d4929922e97b4d9725de0" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46833 } } peers { permanent_uuid: "a57bde86740e4fdaaf06588405ec2549" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 35465 } }
I20260806 00:58:52.135871 3050 raft_consensus.cc:374] T 29aab19c1faa4202ae3b43da5f7088c1 P a57bde86740e4fdaaf06588405ec2549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:52.135892 3050 raft_consensus.cc:729] T 29aab19c1faa4202ae3b43da5f7088c1 P a57bde86740e4fdaaf06588405ec2549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57bde86740e4fdaaf06588405ec2549, State: Initialized, Role: FOLLOWER
I20260806 00:58:52.136018 3050 consensus_queue.cc:260] T 29aab19c1faa4202ae3b43da5f7088c1 P a57bde86740e4fdaaf06588405ec2549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "163d0412a9904986999c9c19bffb4aa1" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38843 } } peers { permanent_uuid: "5b79d6af945d4929922e97b4d9725de0" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46833 } } peers { permanent_uuid: "a57bde86740e4fdaaf06588405ec2549" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 35465 } }
I20260806 00:58:52.136312 3050 ts_tablet_manager.cc:1428] T 29aab19c1faa4202ae3b43da5f7088c1 P a57bde86740e4fdaaf06588405ec2549: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 00:58:52.136322 3055 tablet_bootstrap.cc:492] T 29aab19c1faa4202ae3b43da5f7088c1 P 163d0412a9904986999c9c19bffb4aa1: Bootstrap starting.
I20260806 00:58:52.136442 3033 heartbeater.cc:503] Master 127.23.242.62:38759 was elected leader, sending a full tablet report...
I20260806 00:58:52.136933 3055 tablet_bootstrap.cc:654] T 29aab19c1faa4202ae3b43da5f7088c1 P 163d0412a9904986999c9c19bffb4aa1: Neither blocks nor log segments found. Creating new log.
I20260806 00:58:52.136893 3051 raft_consensus.cc:348] T 29aab19c1faa4202ae3b43da5f7088c1 P 5b79d6af945d4929922e97b4d9725de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "163d0412a9904986999c9c19bffb4aa1" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38843 } } peers { permanent_uuid: "5b79d6af945d4929922e97b4d9725de0" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46833 } } peers { permanent_uuid: "a57bde86740e4fdaaf06588405ec2549" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 35465 } }
I20260806 00:58:52.137030 3051 raft_consensus.cc:374] T 29aab19c1faa4202ae3b43da5f7088c1 P 5b79d6af945d4929922e97b4d9725de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:52.137056 3051 raft_consensus.cc:729] T 29aab19c1faa4202ae3b43da5f7088c1 P 5b79d6af945d4929922e97b4d9725de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b79d6af945d4929922e97b4d9725de0, State: Initialized, Role: FOLLOWER
I20260806 00:58:52.137171 3051 consensus_queue.cc:260] T 29aab19c1faa4202ae3b43da5f7088c1 P 5b79d6af945d4929922e97b4d9725de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "163d0412a9904986999c9c19bffb4aa1" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38843 } } peers { permanent_uuid: "5b79d6af945d4929922e97b4d9725de0" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46833 } } peers { permanent_uuid: "a57bde86740e4fdaaf06588405ec2549" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 35465 } }
I20260806 00:58:52.137459 3051 ts_tablet_manager.cc:1428] T 29aab19c1faa4202ae3b43da5f7088c1 P 5b79d6af945d4929922e97b4d9725de0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 00:58:52.137490 2763 heartbeater.cc:503] Master 127.23.242.62:38759 was elected leader, sending a full tablet report...
I20260806 00:58:52.137741 3055 log.cc:824] T 29aab19c1faa4202ae3b43da5f7088c1 P 163d0412a9904986999c9c19bffb4aa1: Log is configured to *not* fsync() on all Append() calls
I20260806 00:58:52.138664 3055 tablet_bootstrap.cc:492] T 29aab19c1faa4202ae3b43da5f7088c1 P 163d0412a9904986999c9c19bffb4aa1: No bootstrap required, opened a new log
I20260806 00:58:52.138741 3055 ts_tablet_manager.cc:1397] T 29aab19c1faa4202ae3b43da5f7088c1 P 163d0412a9904986999c9c19bffb4aa1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 00:58:52.140221 3055 raft_consensus.cc:348] T 29aab19c1faa4202ae3b43da5f7088c1 P 163d0412a9904986999c9c19bffb4aa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "163d0412a9904986999c9c19bffb4aa1" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38843 } } peers { permanent_uuid: "5b79d6af945d4929922e97b4d9725de0" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46833 } } peers { permanent_uuid: "a57bde86740e4fdaaf06588405ec2549" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 35465 } }
I20260806 00:58:52.140415 3055 raft_consensus.cc:374] T 29aab19c1faa4202ae3b43da5f7088c1 P 163d0412a9904986999c9c19bffb4aa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:58:52.140465 3055 raft_consensus.cc:729] T 29aab19c1faa4202ae3b43da5f7088c1 P 163d0412a9904986999c9c19bffb4aa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 163d0412a9904986999c9c19bffb4aa1, State: Initialized, Role: FOLLOWER
I20260806 00:58:52.140591 3055 consensus_queue.cc:260] T 29aab19c1faa4202ae3b43da5f7088c1 P 163d0412a9904986999c9c19bffb4aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "163d0412a9904986999c9c19bffb4aa1" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38843 } } peers { permanent_uuid: "5b79d6af945d4929922e97b4d9725de0" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46833 } } peers { permanent_uuid: "a57bde86740e4fdaaf06588405ec2549" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 35465 } }
I20260806 00:58:52.140904 3055 ts_tablet_manager.cc:1428] T 29aab19c1faa4202ae3b43da5f7088c1 P 163d0412a9904986999c9c19bffb4aa1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 00:58:52.140951 2898 heartbeater.cc:503] Master 127.23.242.62:38759 was elected leader, sending a full tablet report...
I20260806 00:58:52.144645 3056 raft_consensus.cc:482] T 29aab19c1faa4202ae3b43da5f7088c1 P 5b79d6af945d4929922e97b4d9725de0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 00:58:52.144788 3056 raft_consensus.cc:504] T 29aab19c1faa4202ae3b43da5f7088c1 P 5b79d6af945d4929922e97b4d9725de0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "163d0412a9904986999c9c19bffb4aa1" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38843 } } peers { permanent_uuid: "5b79d6af945d4929922e97b4d9725de0" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46833 } } peers { permanent_uuid: "a57bde86740e4fdaaf06588405ec2549" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 35465 } }
I20260806 00:58:52.145259 3056 leader_election.cc:288] T 29aab19c1faa4202ae3b43da5f7088c1 P 5b79d6af945d4929922e97b4d9725de0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 163d0412a9904986999c9c19bffb4aa1 (127.23.242.2:38843), a57bde86740e4fdaaf06588405ec2549 (127.23.242.3:35465)
I20260806 00:58:52.149116 3049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:52.145873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:38397 (local address 127.23.242.3:35465)
0806 00:58:52.145950 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:52.145953 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:52.145962 (+ 9us) server_negotiation.cc:407] Connection header received
0806 00:58:52.145989 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:52.145990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:52.146000 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:52.146067 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:52.146229 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.146923 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.148263 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.148703 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.148802 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:52.148813 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:52.148992 (+ 179us) server_negotiation.cc:299] Negotiation successful
0806 00:58:52.149035 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 00:58:52.149128 3047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:52.145791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:58403 (local address 127.23.242.2:38843)
0806 00:58:52.145842 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:52.145844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:52.145851 (+ 7us) server_negotiation.cc:407] Connection header received
0806 00:58:52.145873 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:52.145874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:52.145886 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:52.145964 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:58:52.146202 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.146937 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.148237 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.148695 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.148807 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:52.148816 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:52.149005 (+ 189us) server_negotiation.cc:299] Negotiation successful
0806 00:58:52.149052 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260806 00:58:52.149137 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:52.145577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:38843 (local address 127.23.242.1:58403)
0806 00:58:52.145667 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:52.145681 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:52.145738 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:52.146020 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:52.146021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:52.146023 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:52.146106 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:52.146108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.147037 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.147039 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:52.148109 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:52.148113 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.148807 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.148808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:52.148906 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:52.148909 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 00:58:52.148994 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 00:58:52.149042 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":32}
I20260806 00:58:52.149139 3059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:52.145577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:35465 (local address 127.23.242.1:38397)
0806 00:58:52.145832 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:52.145846 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 00:58:52.145912 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:58:52.146069 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:58:52.146069 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:58:52.146071 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:58:52.146138 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:52.146139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.147016 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.147017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:52.148112 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:58:52.148115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.148802 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.148804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:58:52.148892 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:52.148896 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 00:58:52.148982 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 00:58:52.149028 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":199,"thread_start_us":135,"threads_started":1}
I20260806 00:58:52.149546 2852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29aab19c1faa4202ae3b43da5f7088c1" candidate_uuid: "5b79d6af945d4929922e97b4d9725de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "163d0412a9904986999c9c19bffb4aa1" is_pre_election: true
I20260806 00:58:52.149734 2852 raft_consensus.cc:2454] T 29aab19c1faa4202ae3b43da5f7088c1 P 163d0412a9904986999c9c19bffb4aa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b79d6af945d4929922e97b4d9725de0 in term 0.
I20260806 00:58:52.150080 2650 leader_election.cc:302] T 29aab19c1faa4202ae3b43da5f7088c1 P 5b79d6af945d4929922e97b4d9725de0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 163d0412a9904986999c9c19bffb4aa1, 5b79d6af945d4929922e97b4d9725de0; no voters:
I20260806 00:58:52.150255 3056 raft_consensus.cc:2781] T 29aab19c1faa4202ae3b43da5f7088c1 P 5b79d6af945d4929922e97b4d9725de0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 00:58:52.150803 3056 raft_consensus.cc:482] T 29aab19c1faa4202ae3b43da5f7088c1 P 5b79d6af945d4929922e97b4d9725de0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 00:58:52.150870 3056 raft_consensus.cc:3037] T 29aab19c1faa4202ae3b43da5f7088c1 P 5b79d6af945d4929922e97b4d9725de0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:52.151777 3056 raft_consensus.cc:504] T 29aab19c1faa4202ae3b43da5f7088c1 P 5b79d6af945d4929922e97b4d9725de0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "163d0412a9904986999c9c19bffb4aa1" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38843 } } peers { permanent_uuid: "5b79d6af945d4929922e97b4d9725de0" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46833 } } peers { permanent_uuid: "a57bde86740e4fdaaf06588405ec2549" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 35465 } }
I20260806 00:58:52.151548 2987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29aab19c1faa4202ae3b43da5f7088c1" candidate_uuid: "5b79d6af945d4929922e97b4d9725de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57bde86740e4fdaaf06588405ec2549" is_pre_election: true
I20260806 00:58:52.151937 3056 leader_election.cc:288] T 29aab19c1faa4202ae3b43da5f7088c1 P 5b79d6af945d4929922e97b4d9725de0 [CANDIDATE]: Term 1 election: Requested vote from peers 163d0412a9904986999c9c19bffb4aa1 (127.23.242.2:38843), a57bde86740e4fdaaf06588405ec2549 (127.23.242.3:35465)
I20260806 00:58:52.152210 2987 raft_consensus.cc:2454] T 29aab19c1faa4202ae3b43da5f7088c1 P a57bde86740e4fdaaf06588405ec2549 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b79d6af945d4929922e97b4d9725de0 in term 0.
I20260806 00:58:52.152212 2852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29aab19c1faa4202ae3b43da5f7088c1" candidate_uuid: "5b79d6af945d4929922e97b4d9725de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "163d0412a9904986999c9c19bffb4aa1"
I20260806 00:58:52.152302 2852 raft_consensus.cc:3037] T 29aab19c1faa4202ae3b43da5f7088c1 P 163d0412a9904986999c9c19bffb4aa1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:52.152608 2987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29aab19c1faa4202ae3b43da5f7088c1" candidate_uuid: "5b79d6af945d4929922e97b4d9725de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57bde86740e4fdaaf06588405ec2549"
I20260806 00:58:52.152702 2987 raft_consensus.cc:3037] T 29aab19c1faa4202ae3b43da5f7088c1 P a57bde86740e4fdaaf06588405ec2549 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:58:52.153205 2852 raft_consensus.cc:2454] T 29aab19c1faa4202ae3b43da5f7088c1 P 163d0412a9904986999c9c19bffb4aa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b79d6af945d4929922e97b4d9725de0 in term 1.
I20260806 00:58:52.153430 2650 leader_election.cc:302] T 29aab19c1faa4202ae3b43da5f7088c1 P 5b79d6af945d4929922e97b4d9725de0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 163d0412a9904986999c9c19bffb4aa1, 5b79d6af945d4929922e97b4d9725de0; no voters:
I20260806 00:58:52.153571 2987 raft_consensus.cc:2454] T 29aab19c1faa4202ae3b43da5f7088c1 P a57bde86740e4fdaaf06588405ec2549 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b79d6af945d4929922e97b4d9725de0 in term 1.
I20260806 00:58:52.153586 3056 raft_consensus.cc:2781] T 29aab19c1faa4202ae3b43da5f7088c1 P 5b79d6af945d4929922e97b4d9725de0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:58:52.153793 3056 raft_consensus.cc:686] T 29aab19c1faa4202ae3b43da5f7088c1 P 5b79d6af945d4929922e97b4d9725de0 [term 1 LEADER]: Becoming Leader. State: Replica: 5b79d6af945d4929922e97b4d9725de0, State: Running, Role: LEADER
I20260806 00:58:52.153934 3056 consensus_queue.cc:237] T 29aab19c1faa4202ae3b43da5f7088c1 P 5b79d6af945d4929922e97b4d9725de0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "163d0412a9904986999c9c19bffb4aa1" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38843 } } peers { permanent_uuid: "5b79d6af945d4929922e97b4d9725de0" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46833 } } peers { permanent_uuid: "a57bde86740e4fdaaf06588405ec2549" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 35465 } }
I20260806 00:58:52.155170 2570 catalog_manager.cc:5495] T 29aab19c1faa4202ae3b43da5f7088c1 P 5b79d6af945d4929922e97b4d9725de0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b79d6af945d4929922e97b4d9725de0 (127.23.242.1). New cstate: current_term: 1 leader_uuid: "5b79d6af945d4929922e97b4d9725de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "163d0412a9904986999c9c19bffb4aa1" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b79d6af945d4929922e97b4d9725de0" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a57bde86740e4fdaaf06588405ec2549" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 35465 } health_report { overall_health: UNKNOWN } } }
I20260806 00:58:52.165958 3048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:52.162184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58628 (local address 127.23.242.1:46833)
0806 00:58:52.162291 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 00:58:52.162292 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 00:58:52.162301 (+ 9us) server_negotiation.cc:407] Connection header received
0806 00:58:52.162340 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:58:52.162341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:58:52.162364 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:58:52.162436 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 00:58:52.163069 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.163663 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.164614 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:58:52.164831 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:58:52.164895 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:58:52.165374 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 00:58:52.165490 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 00:58:52.165568 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 00:58:52.165847 (+ 279us) server_negotiation.cc:299] Negotiation successful
0806 00:58:52.165873 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 00:58:52.168682 2852 raft_consensus.cc:1261] T 29aab19c1faa4202ae3b43da5f7088c1 P 163d0412a9904986999c9c19bffb4aa1 [term 1 FOLLOWER]: Refusing update from remote peer 5b79d6af945d4929922e97b4d9725de0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:52.168722 2987 raft_consensus.cc:1261] T 29aab19c1faa4202ae3b43da5f7088c1 P a57bde86740e4fdaaf06588405ec2549 [term 1 FOLLOWER]: Refusing update from remote peer 5b79d6af945d4929922e97b4d9725de0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:58:52.169369 3060 consensus_queue.cc:1035] T 29aab19c1faa4202ae3b43da5f7088c1 P 5b79d6af945d4929922e97b4d9725de0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "163d0412a9904986999c9c19bffb4aa1" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:52.169672 3056 consensus_queue.cc:1035] T 29aab19c1faa4202ae3b43da5f7088c1 P 5b79d6af945d4929922e97b4d9725de0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a57bde86740e4fdaaf06588405ec2549" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 35465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:58:52.173482 3063 mvcc.cc:205] Tried to move back new op lower bound from 7315365610159882240 to 7315365610103427072. Current Snapshot: MvccSnapshot[applied={T|T < 7315365610159882240}]
I20260806 00:58:52.174460 3065 mvcc.cc:205] Tried to move back new op lower bound from 7315365610159882240 to 7315365610103427072. Current Snapshot: MvccSnapshot[applied={T|T < 7315365610159882240}]
I20260806 00:58:52.175858 2569 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:58:52.175976 2569 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:58:52.176988 2569 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74f2202f01f84b03af55b671b2afb10d: Sending DeleteTablet for 3 replicas of tablet 29aab19c1faa4202ae3b43da5f7088c1
I20260806 00:58:52.177390 2697 tablet_service.cc:1514] Processing DeleteTablet for tablet 29aab19c1faa4202ae3b43da5f7088c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:52 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:58624
I20260806 00:58:52.177448 2832 tablet_service.cc:1514] Processing DeleteTablet for tablet 29aab19c1faa4202ae3b43da5f7088c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:52 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:51874
I20260806 00:58:52.177444 2967 tablet_service.cc:1514] Processing DeleteTablet for tablet 29aab19c1faa4202ae3b43da5f7088c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:58:52 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:55732
I20260806 00:58:52.177742 3074 tablet_replica.cc:330] T 29aab19c1faa4202ae3b43da5f7088c1 P a57bde86740e4fdaaf06588405ec2549: stopping tablet replica
I20260806 00:58:52.177861 3074 raft_consensus.cc:2229] T 29aab19c1faa4202ae3b43da5f7088c1 P a57bde86740e4fdaaf06588405ec2549 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:52.177973 3074 raft_consensus.cc:2258] T 29aab19c1faa4202ae3b43da5f7088c1 P a57bde86740e4fdaaf06588405ec2549 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:52.178082 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 2633
I20260806 00:58:52.178153 3076 tablet_replica.cc:330] T 29aab19c1faa4202ae3b43da5f7088c1 P 163d0412a9904986999c9c19bffb4aa1: stopping tablet replica
I20260806 00:58:52.178254 3076 raft_consensus.cc:2229] T 29aab19c1faa4202ae3b43da5f7088c1 P 163d0412a9904986999c9c19bffb4aa1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:58:52.178350 3076 raft_consensus.cc:2258] T 29aab19c1faa4202ae3b43da5f7088c1 P 163d0412a9904986999c9c19bffb4aa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:58:52.178954 3076 ts_tablet_manager.cc:1905] T 29aab19c1faa4202ae3b43da5f7088c1 P 163d0412a9904986999c9c19bffb4aa1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:52.179874 3074 ts_tablet_manager.cc:1905] T 29aab19c1faa4202ae3b43da5f7088c1 P a57bde86740e4fdaaf06588405ec2549: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:58:52.181115 3076 ts_tablet_manager.cc:1918] T 29aab19c1faa4202ae3b43da5f7088c1 P 163d0412a9904986999c9c19bffb4aa1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 00:58:52.181208 3076 log.cc:1196] T 29aab19c1faa4202ae3b43da5f7088c1 P 163d0412a9904986999c9c19bffb4aa1: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-1/wal/wals/29aab19c1faa4202ae3b43da5f7088c1
I20260806 00:58:52.181528 3076 ts_tablet_manager.cc:1939] T 29aab19c1faa4202ae3b43da5f7088c1 P 163d0412a9904986999c9c19bffb4aa1: Deleting consensus metadata
I20260806 00:58:52.181866 3074 ts_tablet_manager.cc:1918] T 29aab19c1faa4202ae3b43da5f7088c1 P a57bde86740e4fdaaf06588405ec2549: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 00:58:52.181969 3074 log.cc:1196] T 29aab19c1faa4202ae3b43da5f7088c1 P a57bde86740e4fdaaf06588405ec2549: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785977869987768-24520-0/minicluster-data/ts-2/wal/wals/29aab19c1faa4202ae3b43da5f7088c1
I20260806 00:58:52.182279 3074 ts_tablet_manager.cc:1939] T 29aab19c1faa4202ae3b43da5f7088c1 P a57bde86740e4fdaaf06588405ec2549: Deleting consensus metadata
I20260806 00:58:52.182454 2554 catalog_manager.cc:4841] TS 163d0412a9904986999c9c19bffb4aa1 (127.23.242.2:38843): tablet 29aab19c1faa4202ae3b43da5f7088c1 (table test-table [id=686bf5aa3c94449aa2b1b5220276f9e4]) successfully deleted
I20260806 00:58:52.183008 2557 catalog_manager.cc:4841] TS a57bde86740e4fdaaf06588405ec2549 (127.23.242.3:35465): tablet 29aab19c1faa4202ae3b43da5f7088c1 (table test-table [id=686bf5aa3c94449aa2b1b5220276f9e4]) successfully deleted
W20260806 00:58:52.186003 2555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.242.1:46833 (error 108)
I20260806 00:58:52.186233 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 2768
I20260806 00:58:52.190944 3044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:58:52.190769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:46833 (local address 127.0.0.1:58640)
0806 00:58:52.190845 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0806 00:58:52.190905 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.1:46833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 00:58:52.191112 2555 catalog_manager.cc:4584] TS 5b79d6af945d4929922e97b4d9725de0 (127.23.242.1:46833): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 29aab19c1faa4202ae3b43da5f7088c1: Network error: Client connection negotiation failed: client connection to 127.23.242.1:46833: connect: Connection refused (error 111)
I20260806 00:58:52.192950 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 2903
I20260806 00:58:52.199694 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 2538
2026-08-06T00:58:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2153 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 00:58:52 dist-test-slave-dhph krb5kdc[3080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 00:58:52 dist-test-slave-dhph krb5kdc[3080](info): set up 2 sockets
Aug 06 00:58:52 dist-test-slave-dhph krb5kdc[3080](info): commencing operation
krb5kdc: starting...
W20260806 00:58:53.333668 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 00:58:53 dist-test-slave-dhph krb5kdc[3080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T00:58:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T00:58:53Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.242.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.242.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.242.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/rangeradmin_127.23.242.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.242.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/rangeradmin_127.23.242.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.242.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.242.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.242.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/rangerlookup_127.23.242.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.242.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/rangerlookup_127.23.242.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.242.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.242.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.242.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/HTTP_127.23.242.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.242.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/HTTP_127.23.242.20@KRBTEST.COM.keytab.
I20260806 00:58:53.540014 24520 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskh8enal/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/postgres -l logfile start
2026-08-06 00:58:57.051 UTC [3110] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 00:58:57.051 UTC [3110] LOG: listening on IPv4 address "127.23.242.20", port 54403
2026-08-06 00:58:57.057 UTC [3110] LOG: listening on Unix socket "/tmp/.s.PGSQL.54403"
2026-08-06 00:58:57.098 UTC [3113] LOG: database system was shut down at 2026-08-06 00:58:54 UTC
2026-08-06 00:58:57.105 UTC [3110] LOG: database system is ready to accept connections
I20260806 00:58:58.163421 24520 mini_postgres.cc:96] Postgres bound to 54403
2026-08-06 00:58:58.173 UTC [3121] FATAL: database "slave" does not exist
127.23.242.20:54403 - accepting connections
I20260806 00:58:58.174329 24520 mini_ranger.cc:161] Starting Ranger...
I20260806 00:58:58.194243 24520 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 00:58:58.954633 24520 mini_ranger.cc:87] Created ranger Postgres database
I20260806 00:58:58.954739 24520 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ranger-admin
2026-08-06 00:58:59,305 [I] DB FLAVOR :POSTGRES
2026-08-06 00:58:59,305 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 00:58:59,305 [I] Checking connection..
2026-08-06 00:58:59,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:54403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 00:58:59,588 [I] Checking connection passed.
2026-08-06 00:58:59,589 [I] --------- Verifying version history table ---------
2026-08-06 00:58:59,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:54403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 00:58:59,880 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 00:58:59,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:54403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 00:59:00,159 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 00:59:00,159 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 00:59:00,160 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:54403/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 00:59:00.424 UTC [3185] WARNING: there is no transaction in progress
2026-08-06 00:59:00,432 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 00:59:00,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:54403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 00:59:00,722 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 00:59:00,723 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 00:59:00,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:54403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 00:59:00,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:54403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 00:59:01,277 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:54403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dhph.c.gcp-upstream.internal','N') ;"
2026-08-06 00:59:01,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:54403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 00:59:01,831 [I] Table x_portal_user does not exist in database ranger
2026-08-06 00:59:01,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:54403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 00:59:02,148 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 00:59:02,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:54403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 00:59:02,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:54403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 00:59:02,717 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 00:59:02,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:54403/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 00:59:04.088 UTC [3296] WARNING: there is no transaction in progress
2026-08-06 00:59:04.112 UTC [3296] WARNING: there is no transaction in progress
2026-08-06 00:59:04.138 UTC [3296] WARNING: there is no transaction in progress
2026-08-06 00:59:04.165 UTC [3296] WARNING: there is no transaction in progress
2026-08-06 00:59:04.187 UTC [3296] WARNING: there is no transaction in progress
2026-08-06 00:59:04.209 UTC [3296] WARNING: there is no transaction in progress
2026-08-06 00:59:04.284 UTC [3296] WARNING: there is no transaction in progress
2026-08-06 00:59:04.294 UTC [3296] WARNING: there is no transaction in progress
2026-08-06 00:59:04.315 UTC [3296] WARNING: there is no transaction in progress
2026-08-06 00:59:04.328 UTC [3296] WARNING: there is no transaction in progress
2026-08-06 00:59:04.341 UTC [3296] WARNING: there is no transaction in progress
2026-08-06 00:59:04.360 UTC [3296] WARNING: there is no transaction in progress
2026-08-06 00:59:04.374 UTC [3296] WARNING: there is no transaction in progress
2026-08-06 00:59:04.385 UTC [3296] WARNING: there is no transaction in progress
2026-08-06 00:59:04.397 UTC [3296] WARNING: there is no transaction in progress
2026-08-06 00:59:04.408 UTC [3296] WARNING: there is no transaction in progress
2026-08-06 00:59:05,430 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 00:59:05,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:54403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 00:59:05,724 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 00:59:05,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:54403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 00:59:06,045 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 00:59:06,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:54403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 00:59:06,356 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 00:59:06,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:54403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 00:59:06,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:54403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 00:59:06,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:54403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dhph.c.gcp-upstream.internal';"
2026-08-06 00:59:07,275 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 00:59:07,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:54403/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 00:59:07,575 [I] DB_PATCHES have already been applied
I20260806 00:59:07.582034 24520 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ranger-admin:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskh8enal/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 00:59:07.582116 24520 mini_ranger.cc:193] Using host: 127.23.242.20
I20260806 00:59:07.586139 24520 mini_ranger.cc:227] Ranger admin URL: http://127.23.242.20:46077
Aug 06, 2026 12:59:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 12:59:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 12:59:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 12:59:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 12:59:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 12:59:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 12:59:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 00:59:08.838362 24520 mini_ranger.cc:160] Time spent starting Ranger: real 10.664s user 0.000s sys 0.013s
Aug 06, 2026 12:59:34 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 12:59:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 06, 2026 12:59:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 12:59:34 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 12:59:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 12:59:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 12:59:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 12:59:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 12:59:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 12:59:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 12:59:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 12:59:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 12:59:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 12:59:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 12:59:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 12:59:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 12:59:34 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 12:59:35 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 00:59:36.234705 24520 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.242.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.242.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.242.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/rangerkms_127.23.242.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.242.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/rangerkms_127.23.242.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.242.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.242.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.242.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/HTTP_127.23.242.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.242.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/HTTP_127.23.242.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 00:59:36 dist-test-slave-dhph krb5kdc[3080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977976, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 00:59:36.348924 24520 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 00:59:36.366292 24520 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 00:59:37.355731 24520 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 00:59:37.355847 24520 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ranger-kms
I20260806 00:59:37.355861 24520 mini_ranger_kms.cc:227] Using postgres at 127.23.242.20:54403
2026-08-06 00:59:38,049 [I] DB FLAVOR :POSTGRES
2026-08-06 00:59:38,049 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 00:59:38,049 [I] Checking connection
2026-08-06 00:59:38,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:54403/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 00:59:38,362 [I] connection success
2026-08-06 00:59:38,362 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 00:59:38,362 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 00:59:38,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:54403/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 00:59:38,643 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 00:59:38,643 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 00:59:38,644 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 00:59:38,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:54403/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 00:59:38,955 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 00:59:39.511590 24520 mini_ranger_kms.cc:310] Created kms service
I20260806 00:59:39.728135 24520 mini_ranger_kms.cc:326] Created kudu user
I20260806 00:59:39.786445 24520 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 00:59:39 dist-test-slave-dhph krb5kdc[3080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977976, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.242.20@KRBTEST.COM
I20260806 00:59:40.300417 24520 mini_ranger_kms.cc:384] Added ranger policy
I20260806 00:59:40.300545 24520 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ranger-kms:/tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 00:59:40.300570 24520 mini_ranger_kms.cc:240] Using host: 127.23.242.20
I20260806 00:59:40.305198 24520 mini_ranger_kms.cc:276] Ranger KMS PID: 3483
I20260806 00:59:40.305361 24520 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.242.20:55711
Aug 06, 2026 12:59:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 12:59:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 12:59:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 12:59:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 12:59:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 12:59:42 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260806 00:59:42.980773 24520 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.632s user 0.012s sys 0.325s
I20260806 00:59:42.980898 24520 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 00:59:42.980991 24520 mini_ranger_kms.cc:401] 127.23.242.20:55711/kms/v1/keys
Aug 06 00:59:43 dist-test-slave-dhph krb5kdc[3080](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785977983, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.242.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 06 00:59:43 dist-test-slave-dhph krb5kdc[3080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977983, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.242.20@KRBTEST.COM for HTTP/127.23.242.20@KRBTEST.COM
Aug 06 00:59:43 dist-test-slave-dhph krb5kdc[3080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977983, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.242.20@KRBTEST.COM for HTTP/127.23.242.20@KRBTEST.COM
Aug 06 00:59:46 dist-test-slave-dhph krb5kdc[3080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 00:59:46.774662 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:44247
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:45475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:44247
--ranger_config_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.242.20:55711/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:59:46.890561 3533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:59:46.890887 3533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:59:46.890933 3533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:59:46.892860 3533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 00:59:46.892956 3533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:59:46.892980 3533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:59:46.893000 3533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 00:59:46.893033 3533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 00:59:46.895462 3533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:45475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.242.20:55711/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:44247
--ranger_config_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:44247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:59:46.895887 3533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:59:46.896214 3533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 00:59:46.902910 3533 server_base.cc:1034] running on GCE node
W20260806 00:59:46.903093 3538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:59:46.903000 3539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:59:46.903153 3541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:59:46.903613 3533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:59:46.904048 3533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:59:46.905237 3533 hybrid_clock.cc:648] HybridClock initialized: now 1785977986905210 us; error 45 us; skew 500 ppm
Aug 06 00:59:46 dist-test-slave-dhph krb5kdc[3080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977986, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:59:46.914407 3533 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 00:59:46.915318 3533 webserver.cc:460] Webserver started at http://127.23.242.62:33671/ using document root <none> and password file <none>
I20260806 00:59:46.915639 3533 fs_manager.cc:359] Metadata directory not provided
I20260806 00:59:46.915848 3533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:59:46.916076 3533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 00:59:46 dist-test-slave-dhph krb5kdc[3080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977986, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for HTTP/127.23.242.20@KRBTEST.COM
I20260806 00:59:46.971443 3533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "a86e83b51d95430791ea4834db586e31"
format_stamp: "Formatted at 2026-08-06 00:59:46 on dist-test-slave-dhph"
server_key: "8ab3143060f6bf2de44d65af230f29c4"
server_key_iv: "df4736ce5266925a62f89be791a0757f"
server_key_version: "kuduclusterkey@0"
I20260806 00:59:46.972000 3533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "a86e83b51d95430791ea4834db586e31"
format_stamp: "Formatted at 2026-08-06 00:59:46 on dist-test-slave-dhph"
server_key: "8ab3143060f6bf2de44d65af230f29c4"
server_key_iv: "df4736ce5266925a62f89be791a0757f"
server_key_version: "kuduclusterkey@0"
I20260806 00:59:46.974340 3533 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260806 00:59:46.988294 3548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:59:46.988576 3533 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 00:59:46.988695 3533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "a86e83b51d95430791ea4834db586e31"
format_stamp: "Formatted at 2026-08-06 00:59:46 on dist-test-slave-dhph"
server_key: "8ab3143060f6bf2de44d65af230f29c4"
server_key_iv: "df4736ce5266925a62f89be791a0757f"
server_key_version: "kuduclusterkey@0"
I20260806 00:59:46.988819 3533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:59:47.009608 3533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:59:47.012009 3533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:59:47.012274 3533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:59:47.024279 3533 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:44247
I20260806 00:59:47.024462 3601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:44247 every 8 connection(s)
I20260806 00:59:47.024966 3533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 00:59:47.026230 3602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:59:47.029678 3602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a86e83b51d95430791ea4834db586e31: Bootstrap starting.
I20260806 00:59:47.030432 3602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a86e83b51d95430791ea4834db586e31: Neither blocks nor log segments found. Creating new log.
I20260806 00:59:47.030843 3602 log.cc:824] T 00000000000000000000000000000000 P a86e83b51d95430791ea4834db586e31: Log is configured to *not* fsync() on all Append() calls
I20260806 00:59:47.031770 3602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a86e83b51d95430791ea4834db586e31: No bootstrap required, opened a new log
I20260806 00:59:47.034217 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 3533
I20260806 00:59:47.034335 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 00:59:47.034333 3602 raft_consensus.cc:348] T 00000000000000000000000000000000 P a86e83b51d95430791ea4834db586e31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a86e83b51d95430791ea4834db586e31" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44247 } }
I20260806 00:59:47.034544 24520 external_mini_cluster.cc:1428] Setting key a0993e1a4adc9507ce674f85092503ee
I20260806 00:59:47.034567 3602 raft_consensus.cc:374] T 00000000000000000000000000000000 P a86e83b51d95430791ea4834db586e31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:59:47.034643 3602 raft_consensus.cc:729] T 00000000000000000000000000000000 P a86e83b51d95430791ea4834db586e31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a86e83b51d95430791ea4834db586e31, State: Initialized, Role: FOLLOWER
I20260806 00:59:47.035266 3602 consensus_queue.cc:260] T 00000000000000000000000000000000 P a86e83b51d95430791ea4834db586e31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a86e83b51d95430791ea4834db586e31" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44247 } }
I20260806 00:59:47.035459 3602 raft_consensus.cc:388] T 00000000000000000000000000000000 P a86e83b51d95430791ea4834db586e31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 00:59:47.035575 3602 raft_consensus.cc:482] T 00000000000000000000000000000000 P a86e83b51d95430791ea4834db586e31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 00:59:47.035655 3602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a86e83b51d95430791ea4834db586e31 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:59:47.036710 3602 raft_consensus.cc:504] T 00000000000000000000000000000000 P a86e83b51d95430791ea4834db586e31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a86e83b51d95430791ea4834db586e31" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44247 } }
I20260806 00:59:47.036896 3602 leader_election.cc:302] T 00000000000000000000000000000000 P a86e83b51d95430791ea4834db586e31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a86e83b51d95430791ea4834db586e31; no voters:
I20260806 00:59:47.037168 3602 leader_election.cc:288] T 00000000000000000000000000000000 P a86e83b51d95430791ea4834db586e31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 00:59:47.037556 3608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a86e83b51d95430791ea4834db586e31 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:59:47.037659 3602 sys_catalog.cc:564] T 00000000000000000000000000000000 P a86e83b51d95430791ea4834db586e31 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 00:59:47.037880 3608 raft_consensus.cc:686] T 00000000000000000000000000000000 P a86e83b51d95430791ea4834db586e31 [term 1 LEADER]: Becoming Leader. State: Replica: a86e83b51d95430791ea4834db586e31, State: Running, Role: LEADER
I20260806 00:59:47.037986 3608 consensus_queue.cc:237] T 00000000000000000000000000000000 P a86e83b51d95430791ea4834db586e31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a86e83b51d95430791ea4834db586e31" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44247 } }
I20260806 00:59:47.038439 3608 sys_catalog.cc:455] T 00000000000000000000000000000000 P a86e83b51d95430791ea4834db586e31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a86e83b51d95430791ea4834db586e31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a86e83b51d95430791ea4834db586e31" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44247 } } }
I20260806 00:59:47.038544 3608 sys_catalog.cc:458] T 00000000000000000000000000000000 P a86e83b51d95430791ea4834db586e31 [sys.catalog]: This master's current role is: LEADER
I20260806 00:59:47.038842 3608 sys_catalog.cc:455] T 00000000000000000000000000000000 P a86e83b51d95430791ea4834db586e31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a86e83b51d95430791ea4834db586e31. Latest consensus state: current_term: 1 leader_uuid: "a86e83b51d95430791ea4834db586e31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a86e83b51d95430791ea4834db586e31" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44247 } } }
I20260806 00:59:47.039175 3615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 00:59:47.040186 3608 sys_catalog.cc:458] T 00000000000000000000000000000000 P a86e83b51d95430791ea4834db586e31 [sys.catalog]: This master's current role is: LEADER
I20260806 00:59:47.040782 3615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 00:59:47.040081 3602 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Aug 06 00:59:47 dist-test-slave-dhph krb5kdc[3080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:59:47.047001 3615 catalog_manager.cc:1347] Generated new cluster ID: 4369d6e94bb44832aaaaeef6b59db9bb
I20260806 00:59:47.047076 3615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 00:59:47.050172 3606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:59:47.035261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38244 (local address 127.23.242.62:44247)
0806 00:59:47.035646 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0806 00:59:47.035657 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 00:59:47.035720 (+ 63us) server_negotiation.cc:407] Connection header received
0806 00:59:47.035845 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:59:47.035852 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:59:47.035910 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:59:47.036054 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:59:47.037246 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:59:47.038915 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:59:47.040275 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:59:47.040876 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:59:47.040985 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:59:47.044874 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:59:47.044932 (+ 58us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:59:47.044932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:59:47.045084 (+ 152us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:59:47.048204 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:59:47.048658 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:59:47.048661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:59:47.048661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:59:47.048790 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:59:47.049183 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:59:47.049183 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:59:47.049184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:59:47.049491 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:59:47.049642 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 00:59:47.049783 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 00:59:47.049924 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":204,"threads_started":1}
I20260806 00:59:47.053424 3615 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 00:59:47.054703 3615 catalog_manager.cc:1504] Loading token signing keys...
I20260806 00:59:47.058079 3615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a86e83b51d95430791ea4834db586e31: Generated new TSK 0
I20260806 00:59:47.058298 3615 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.242.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
I20260806 00:59:47.868543 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.242.20:55711/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:0
--local_ip_for_outbound_sockets=127.23.242.1
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:44247
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:45475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:59:48.002852 3650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:59:48.003062 3650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:59:48.003087 3650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:59:48.005199 3650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:59:48.005285 3650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:59:48.005335 3650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 00:59:48.007916 3650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:45475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.242.20:55711/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:44247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:59:48.008329 3650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:59:48.008589 3650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:59:48.015021 3656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:59:48.014986 3655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:59:48.015460 3658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:59:48.015986 3650 server_base.cc:1034] running on GCE node
I20260806 00:59:48.016263 3650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:59:48.016595 3650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:59:48.017827 3650 hybrid_clock.cc:648] HybridClock initialized: now 1785977988017808 us; error 32 us; skew 500 ppm
Aug 06 00:59:48 dist-test-slave-dhph krb5kdc[3080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977988, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:59:48.021054 3650 init.cc:377] Logged in from keytab as kudu/127.23.242.1@KRBTEST.COM (short username kudu)
I20260806 00:59:48.021665 3650 webserver.cc:460] Webserver started at http://127.23.242.1:37689/ using document root <none> and password file <none>
I20260806 00:59:48.022006 3650 fs_manager.cc:359] Metadata directory not provided
I20260806 00:59:48.022104 3650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:59:48.022269 3650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 00:59:48 dist-test-slave-dhph krb5kdc[3080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977988, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for HTTP/127.23.242.20@KRBTEST.COM
I20260806 00:59:48.050338 3650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-0/data/instance:
uuid: "9cc759f9a7814c3791aab1de039bbcad"
format_stamp: "Formatted at 2026-08-06 00:59:48 on dist-test-slave-dhph"
server_key: "29481c9c96d1891b6b992e697aa98a19"
server_key_iv: "728aae168875fe32d7a6d7df8a1efd62"
server_key_version: "kuduclusterkey@0"
I20260806 00:59:48.050931 3650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance:
uuid: "9cc759f9a7814c3791aab1de039bbcad"
format_stamp: "Formatted at 2026-08-06 00:59:48 on dist-test-slave-dhph"
server_key: "29481c9c96d1891b6b992e697aa98a19"
server_key_iv: "728aae168875fe32d7a6d7df8a1efd62"
server_key_version: "kuduclusterkey@0"
I20260806 00:59:48.052708 3650 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260806 00:59:48.082340 3667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:59:48.082660 3650 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260806 00:59:48.082769 3650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "9cc759f9a7814c3791aab1de039bbcad"
format_stamp: "Formatted at 2026-08-06 00:59:48 on dist-test-slave-dhph"
server_key: "29481c9c96d1891b6b992e697aa98a19"
server_key_iv: "728aae168875fe32d7a6d7df8a1efd62"
server_key_version: "kuduclusterkey@0"
I20260806 00:59:48.082895 3650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:59:48.114223 3650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:59:48.117161 3650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:59:48.117434 3650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:59:48.117815 3650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:59:48.118284 3650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:59:48.118381 3650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:59:48.118490 3650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:59:48.118569 3650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:59:48.126907 3650 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:37893
I20260806 00:59:48.127480 3650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 00:59:48.133389 3782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:37893 every 8 connection(s)
I20260806 00:59:48.133764 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 3650
I20260806 00:59:48.133904 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance
I20260806 00:59:48.134164 24520 external_mini_cluster.cc:1428] Setting key 036236b6bcfba33141b304435083a033
Aug 06 00:59:48 dist-test-slave-dhph krb5kdc[3080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977988, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:59:48.145844 3785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:59:48.134652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:46085 (local address 127.23.242.62:44247)
0806 00:59:48.134968 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0806 00:59:48.134971 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 00:59:48.135822 (+ 851us) server_negotiation.cc:407] Connection header received
0806 00:59:48.135847 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:59:48.135848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:59:48.135860 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:59:48.135948 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:59:48.136788 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:59:48.137357 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:59:48.139366 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:59:48.139661 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:59:48.139728 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:59:48.142114 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:59:48.142115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:59:48.142116 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:59:48.142132 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:59:48.144155 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:59:48.144700 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:59:48.144702 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:59:48.144702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:59:48.144775 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:59:48.145063 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:59:48.145064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:59:48.145064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:59:48.145232 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:59:48.145327 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 00:59:48.145632 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 00:59:48.145737 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":83,"threads_started":1}
I20260806 00:59:48.147187 3786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:59:48.134642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:44247 (local address 127.23.242.1:46085)
0806 00:59:48.135042 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0806 00:59:48.135097 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 00:59:48.135690 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:59:48.136035 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:59:48.136040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:59:48.136353 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:59:48.136591 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:59:48.136596 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:59:48.137471 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:59:48.137472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:59:48.139243 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:59:48.139248 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:59:48.139749 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:59:48.139751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:59:48.139884 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:59:48.140546 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:59:48.140556 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:59:48.141993 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:59:48.144279 (+ 2286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:59:48.144290 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:59:48.144293 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:59:48.144603 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:59:48.144847 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:59:48.144848 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:59:48.144848 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:59:48.144992 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:59:48.145330 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:59:48.145337 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:59:48.145490 (+ 153us) client_negotiation.cc:769] Sending connection context
0806 00:59:48.145612 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 00:59:48.146842 (+ 1230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":120,"threads_started":1}
I20260806 00:59:48.148773 3783 heartbeater.cc:347] Connected to a master server at 127.23.242.62:44247
I20260806 00:59:48.148916 3783 heartbeater.cc:464] Registering TS with master...
I20260806 00:59:48.149292 3783 heartbeater.cc:511] Master 127.23.242.62:44247 requested a full tablet report, sending...
I20260806 00:59:48.150378 3565 ts_manager.cc:194] Registered new tserver with Master: 9cc759f9a7814c3791aab1de039bbcad (127.23.242.1:37893)
WARNING: no policy specified for kudu/127.23.242.2@KRBTEST.COM; defaulting to no policy
I20260806 00:59:48.151935 3565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.1@KRBTEST.COM'} at 127.23.242.1:46085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
I20260806 00:59:48.183066 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.242.20:55711/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:0
--local_ip_for_outbound_sockets=127.23.242.2
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:44247
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:45475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 00:59:48 dist-test-slave-dhph krb5kdc[3080](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785977988, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 00:59:48.328701 3789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:59:48.329110 3789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:59:48.329193 3789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:59:48.331794 3789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:59:48.331948 3789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:59:48.332054 3789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 00:59:48.335071 3789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:45475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.242.20:55711/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:44247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:59:48.335638 3789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:59:48.336007 3789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:59:48.344280 3798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:59:48.344399 3797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:59:48.344791 3789 server_base.cc:1034] running on GCE node
W20260806 00:59:48.344964 3800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:59:48.345324 3789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:59:48.345654 3789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:59:48.346844 3789 hybrid_clock.cc:648] HybridClock initialized: now 1785977988346822 us; error 45 us; skew 500 ppm
Aug 06 00:59:48 dist-test-slave-dhph krb5kdc[3080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977988, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:59:48.350481 3789 init.cc:377] Logged in from keytab as kudu/127.23.242.2@KRBTEST.COM (short username kudu)
I20260806 00:59:48.351101 3789 webserver.cc:460] Webserver started at http://127.23.242.2:36509/ using document root <none> and password file <none>
I20260806 00:59:48.351393 3789 fs_manager.cc:359] Metadata directory not provided
I20260806 00:59:48.351462 3789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:59:48.351661 3789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 00:59:48 dist-test-slave-dhph krb5kdc[3080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977988, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for HTTP/127.23.242.20@KRBTEST.COM
I20260806 00:59:48.374990 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-1/data/instance:
uuid: "9423c0b9034d4e64a955cb94b9c9426a"
format_stamp: "Formatted at 2026-08-06 00:59:48 on dist-test-slave-dhph"
server_key: "9b1ff7435fe5c4dac94fdf6f52434a30"
server_key_iv: "22e73a8c7a1476d6e4de39c293055535"
server_key_version: "kuduclusterkey@0"
I20260806 00:59:48.376683 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance:
uuid: "9423c0b9034d4e64a955cb94b9c9426a"
format_stamp: "Formatted at 2026-08-06 00:59:48 on dist-test-slave-dhph"
server_key: "9b1ff7435fe5c4dac94fdf6f52434a30"
server_key_iv: "22e73a8c7a1476d6e4de39c293055535"
server_key_version: "kuduclusterkey@0"
I20260806 00:59:48.383180 3789 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.000s
I20260806 00:59:48.404858 3808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:59:48.405134 3789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 00:59:48.405242 3789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "9423c0b9034d4e64a955cb94b9c9426a"
format_stamp: "Formatted at 2026-08-06 00:59:48 on dist-test-slave-dhph"
server_key: "9b1ff7435fe5c4dac94fdf6f52434a30"
server_key_iv: "22e73a8c7a1476d6e4de39c293055535"
server_key_version: "kuduclusterkey@0"
I20260806 00:59:48.405342 3789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:59:48.428402 3789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:59:48.432238 3789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:59:48.432494 3789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:59:48.432840 3789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:59:48.433461 3789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:59:48.433590 3789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:59:48.433748 3789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:59:48.433941 3789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:59:48.442624 3789 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:45021
I20260806 00:59:48.443243 3789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 00:59:48.444658 3923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:45021 every 8 connection(s)
I20260806 00:59:48.446493 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 3789
I20260806 00:59:48.446633 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance
I20260806 00:59:48.446834 24520 external_mini_cluster.cc:1428] Setting key b135dd6975cfeef0e365f5457869601a
WARNING: no policy specified for kudu/127.23.242.3@KRBTEST.COM; defaulting to no policy
Aug 06 00:59:48 dist-test-slave-dhph krb5kdc[3080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977988, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.3@KRBTEST.COM" created.
I20260806 00:59:48.474738 3785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:59:48.445767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:43667 (local address 127.23.242.62:44247)
0806 00:59:48.445838 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0806 00:59:48.445840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:59:48.446173 (+ 333us) server_negotiation.cc:407] Connection header received
0806 00:59:48.447925 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:59:48.447927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:59:48.447943 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:59:48.447974 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:59:48.449991 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:59:48.450859 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:59:48.452780 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:59:48.453130 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:59:48.453181 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:59:48.466912 (+ 13731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:59:48.466915 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:59:48.466915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:59:48.466926 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:59:48.468849 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:59:48.473054 (+ 4205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:59:48.473056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:59:48.473056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:59:48.473130 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:59:48.473448 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:59:48.473449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:59:48.473449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:59:48.473604 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:59:48.473686 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 00:59:48.474093 (+ 407us) server_negotiation.cc:299] Negotiation successful
0806 00:59:48.474412 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 00:59:48.476480 3926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:59:48.445432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:44247 (local address 127.23.242.2:43667)
0806 00:59:48.446019 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0806 00:59:48.446069 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 00:59:48.447796 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:59:48.448169 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:59:48.448174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:59:48.448463 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:59:48.448941 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:59:48.448949 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:59:48.451116 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:59:48.451117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:59:48.452662 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:59:48.452665 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:59:48.453287 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:59:48.453288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:59:48.453437 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:59:48.461687 (+ 8250us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:59:48.461702 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:59:48.466724 (+ 5022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:59:48.472465 (+ 5741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:59:48.472480 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:59:48.472483 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:59:48.472928 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:59:48.473209 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:59:48.473210 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:59:48.473210 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:59:48.473365 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:59:48.473729 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:59:48.473735 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:59:48.473918 (+ 183us) client_negotiation.cc:769] Sending connection context
0806 00:59:48.475241 (+ 1323us) client_negotiation.cc:240] Negotiation successful
0806 00:59:48.476184 (+ 943us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":95,"threads_started":1}
I20260806 00:59:48.477322 3924 heartbeater.cc:347] Connected to a master server at 127.23.242.62:44247
I20260806 00:59:48.477433 3924 heartbeater.cc:464] Registering TS with master...
I20260806 00:59:48.477659 3924 heartbeater.cc:511] Master 127.23.242.62:44247 requested a full tablet report, sending...
I20260806 00:59:48.478183 3565 ts_manager.cc:194] Registered new tserver with Master: 9423c0b9034d4e64a955cb94b9c9426a (127.23.242.2:45021)
I20260806 00:59:48.478924 3565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.2@KRBTEST.COM'} at 127.23.242.2:43667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
I20260806 00:59:48.498502 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.242.20:55711/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:0
--local_ip_for_outbound_sockets=127.23.242.3
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:44247
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:45475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 00:59:48.662479 3929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 00:59:48.662941 3929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 00:59:48.663062 3929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 00:59:48.665246 3929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 00:59:48.665544 3929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 00:59:48.665688 3929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 00:59:48.668318 3929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:45475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.242.20:55711/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:44247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 00:59:48.668898 3929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 00:59:48.669273 3929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 00:59:48.677086 3937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 00:59:48.677779 3934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:59:48.679143 3929 server_base.cc:1034] running on GCE node
W20260806 00:59:48.681687 3935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 00:59:48.682103 3929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 00:59:48.682458 3929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 00:59:48.683959 3929 hybrid_clock.cc:648] HybridClock initialized: now 1785977988683934 us; error 41 us; skew 500 ppm
Aug 06 00:59:48 dist-test-slave-dhph krb5kdc[3080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977988, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 00:59:48.687822 3929 init.cc:377] Logged in from keytab as kudu/127.23.242.3@KRBTEST.COM (short username kudu)
I20260806 00:59:48.688392 3929 webserver.cc:460] Webserver started at http://127.23.242.3:44169/ using document root <none> and password file <none>
I20260806 00:59:48.688706 3929 fs_manager.cc:359] Metadata directory not provided
I20260806 00:59:48.688769 3929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 00:59:48.688917 3929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 00:59:48 dist-test-slave-dhph krb5kdc[3080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977988, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for HTTP/127.23.242.20@KRBTEST.COM
I20260806 00:59:48.709909 3929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-2/data/instance:
uuid: "556b8380958d45f9bcbf2aa0970f10a4"
format_stamp: "Formatted at 2026-08-06 00:59:48 on dist-test-slave-dhph"
server_key: "602785a6cfbcab6d11215afbea6f5fb2"
server_key_iv: "53d6c4b0136253d7e2152c603c36dc5e"
server_key_version: "kuduclusterkey@0"
I20260806 00:59:48.710647 3929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance:
uuid: "556b8380958d45f9bcbf2aa0970f10a4"
format_stamp: "Formatted at 2026-08-06 00:59:48 on dist-test-slave-dhph"
server_key: "602785a6cfbcab6d11215afbea6f5fb2"
server_key_iv: "53d6c4b0136253d7e2152c603c36dc5e"
server_key_version: "kuduclusterkey@0"
I20260806 00:59:48.717056 3929 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.000s
I20260806 00:59:48.738742 3944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 00:59:48.739086 3929 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260806 00:59:48.739197 3929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "556b8380958d45f9bcbf2aa0970f10a4"
format_stamp: "Formatted at 2026-08-06 00:59:48 on dist-test-slave-dhph"
server_key: "602785a6cfbcab6d11215afbea6f5fb2"
server_key_iv: "53d6c4b0136253d7e2152c603c36dc5e"
server_key_version: "kuduclusterkey@0"
I20260806 00:59:48.739305 3929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 00:59:48.774089 3929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 00:59:48.776847 3929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 00:59:48.777076 3929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 00:59:48.777367 3929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 00:59:48.777839 3929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 00:59:48.777930 3929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 00:59:48.778007 3929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 00:59:48.778071 3929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 00:59:48.790717 3929 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:36391
I20260806 00:59:48.791263 3929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
I20260806 00:59:48.795881 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 3929
I20260806 00:59:48.795992 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance
I20260806 00:59:48.796274 24520 external_mini_cluster.cc:1428] Setting key 4a0daf8ce59681473b0b70d1c0457598
Aug 06 00:59:48 dist-test-slave-dhph krb5kdc[3080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977988, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 00:59:48.802699 4058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:36391 every 8 connection(s)
I20260806 00:59:48.806977 3785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:59:48.792899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:51793 (local address 127.23.242.62:44247)
0806 00:59:48.792959 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0806 00:59:48.792961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:59:48.793209 (+ 248us) server_negotiation.cc:407] Connection header received
0806 00:59:48.793877 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:59:48.793878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:59:48.793890 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:59:48.793920 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:59:48.794903 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:59:48.795635 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:59:48.798952 (+ 3317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:59:48.799239 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:59:48.799284 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:59:48.802259 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:59:48.802261 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:59:48.802261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:59:48.802269 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:59:48.804731 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:59:48.805545 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:59:48.805547 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:59:48.805547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:59:48.805621 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:59:48.805979 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:59:48.805981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:59:48.805981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:59:48.806142 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:59:48.806195 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 00:59:48.806784 (+ 589us) server_negotiation.cc:299] Negotiation successful
0806 00:59:48.806878 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 00:59:48.807801 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:59:48.792869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:44247 (local address 127.23.242.3:51793)
0806 00:59:48.793067 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 00:59:48.793111 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 00:59:48.793753 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:59:48.794097 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:59:48.794101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:59:48.794443 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0806 00:59:48.794795 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:59:48.794801 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:59:48.797257 (+ 2456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:59:48.797258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:59:48.798846 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:59:48.798850 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:59:48.799379 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:59:48.799380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:59:48.799565 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:59:48.800301 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 00:59:48.800313 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 00:59:48.802139 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 00:59:48.804857 (+ 2718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:59:48.804866 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:59:48.804869 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:59:48.805430 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:59:48.805710 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 00:59:48.805711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 00:59:48.805711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 00:59:48.805872 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 00:59:48.806277 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 00:59:48.806283 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 00:59:48.806490 (+ 207us) client_negotiation.cc:769] Sending connection context
0806 00:59:48.807358 (+ 868us) client_negotiation.cc:240] Negotiation successful
0806 00:59:48.807549 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260806 00:59:48.808502 4059 heartbeater.cc:347] Connected to a master server at 127.23.242.62:44247
I20260806 00:59:48.808605 4059 heartbeater.cc:464] Registering TS with master...
I20260806 00:59:48.808796 4059 heartbeater.cc:511] Master 127.23.242.62:44247 requested a full tablet report, sending...
I20260806 00:59:48.809319 3564 ts_manager.cc:194] Registered new tserver with Master: 556b8380958d45f9bcbf2aa0970f10a4 (127.23.242.3:36391)
I20260806 00:59:48.809826 3564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.3@KRBTEST.COM'} at 127.23.242.3:51793
I20260806 00:59:48.827232 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 00:59:48.838696 3785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:59:48.829987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55902 (local address 127.23.242.62:44247)
0806 00:59:48.830034 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 00:59:48.830036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:59:48.830353 (+ 317us) server_negotiation.cc:407] Connection header received
0806 00:59:48.830453 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:59:48.830454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:59:48.830465 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:59:48.830494 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0806 00:59:48.831323 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:59:48.832103 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:59:48.833206 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:59:48.833499 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:59:48.833538 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:59:48.835202 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 00:59:48.835203 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 00:59:48.835204 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 00:59:48.835212 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 00:59:48.837038 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:59:48.837693 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:59:48.837695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:59:48.837695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:59:48.837758 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 00:59:48.837950 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 00:59:48.837951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 00:59:48.837951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 00:59:48.838081 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 00:59:48.838155 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 00:59:48.838365 (+ 210us) server_negotiation.cc:299] Negotiation successful
0806 00:59:48.838491 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 00:59:48.841190 3564 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 00:59:48.844528 3564 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Aug 06 00:59:48 dist-test-slave-dhph krb5kdc[3080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977988, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for HTTP/127.23.242.20@KRBTEST.COM
I20260806 00:59:48.866103 4077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:59:48.848035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:36391 (local address 127.0.0.1:57912)
0806 00:59:48.849454 (+ 1419us) negotiation.cc:107] Waiting for socket to connect
0806 00:59:48.849496 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 00:59:48.849569 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:59:48.849884 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:59:48.849887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:59:48.849890 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:59:48.849965 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:59:48.849966 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:59:48.851668 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:59:48.851669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:59:48.853026 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:59:48.853029 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:59:48.865741 (+ 12712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:59:48.865742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:59:48.865858 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:59:48.865873 (+ 15us) client_negotiation.cc:769] Sending connection context
0806 00:59:48.865942 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 00:59:48.866002 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1386,"thread_start_us":78,"threads_started":1}
I20260806 00:59:48.866839 4075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:59:48.848159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57912 (local address 127.23.242.3:36391)
0806 00:59:48.849295 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0806 00:59:48.849300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 00:59:48.849663 (+ 363us) server_negotiation.cc:407] Connection header received
0806 00:59:48.849712 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:59:48.849714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:59:48.849731 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:59:48.849799 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:59:48.850040 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:59:48.851570 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:59:48.853433 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:59:48.854392 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:59:48.854450 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:59:48.854529 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 00:59:48.866547 (+ 12018us) server_negotiation.cc:299] Negotiation successful
0806 00:59:48.866743 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1097,"thread_start_us":135,"threads_started":1}
I20260806 00:59:48.868582 4076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:59:48.848409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:45021 (local address 127.0.0.1:56722)
0806 00:59:48.855304 (+ 6895us) negotiation.cc:107] Waiting for socket to connect
0806 00:59:48.855315 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:59:48.855566 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:59:48.857578 (+ 2012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:59:48.857579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:59:48.857583 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:59:48.857640 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:59:48.857642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:59:48.861811 (+ 4169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:59:48.861812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:59:48.862934 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:59:48.862936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:59:48.867141 (+ 4205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:59:48.867142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:59:48.868295 (+ 1153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:59:48.868299 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 00:59:48.868453 (+ 154us) client_negotiation.cc:240] Negotiation successful
0806 00:59:48.868509 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6871,"thread_start_us":56,"threads_started":1}
I20260806 00:59:48.868999 4079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:59:48.848617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.23.242.2:45021)
0806 00:59:48.849191 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0806 00:59:48.849197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 00:59:48.855637 (+ 6440us) server_negotiation.cc:407] Connection header received
0806 00:59:48.855699 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:59:48.855702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:59:48.855719 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:59:48.855788 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:59:48.857741 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:59:48.859206 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:59:48.863063 (+ 3857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:59:48.863901 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:59:48.863977 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:59:48.864053 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 00:59:48.868854 (+ 4801us) server_negotiation.cc:299] Negotiation successful
0806 00:59:48.868911 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":96,"threads_started":1}
I20260806 00:59:48.868971 3993 tablet_service.cc:1467] Processing CreateTablet for tablet f796bbdd7f9e4e0d9c797cfaa8ee84f7 (DEFAULT_TABLE table=test-table [id=207a41a359824b7d8c840d4df2b13db9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:59:48.869424 3993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f796bbdd7f9e4e0d9c797cfaa8ee84f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:59:48.869846 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:59:48.848462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48080 (local address 127.23.242.1:37893)
0806 00:59:48.849363 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0806 00:59:48.849367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 00:59:48.854809 (+ 5442us) server_negotiation.cc:407] Connection header received
0806 00:59:48.854856 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:59:48.854859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:59:48.854878 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:59:48.855181 (+ 303us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:59:48.856325 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:59:48.857462 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:59:48.864103 (+ 6641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:59:48.865473 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:59:48.865554 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:59:48.865615 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 00:59:48.869286 (+ 3671us) server_negotiation.cc:299] Negotiation successful
0806 00:59:48.869349 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":874,"thread_start_us":64,"threads_started":1}
I20260806 00:59:48.867961 4074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:59:48.848260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:37893 (local address 127.0.0.1:48080)
0806 00:59:48.854654 (+ 6394us) negotiation.cc:107] Waiting for socket to connect
0806 00:59:48.854667 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 00:59:48.854738 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:59:48.856184 (+ 1446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:59:48.856185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:59:48.856187 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:59:48.856240 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:59:48.856242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:59:48.860417 (+ 4175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:59:48.860418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:59:48.861704 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:59:48.861709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:59:48.867548 (+ 5839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:59:48.867550 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:59:48.867631 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:59:48.867637 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 00:59:48.867805 (+ 168us) client_negotiation.cc:240] Negotiation successful
0806 00:59:48.867865 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6368,"thread_start_us":66,"threads_started":1}
I20260806 00:59:48.870869 3711 tablet_service.cc:1467] Processing CreateTablet for tablet f796bbdd7f9e4e0d9c797cfaa8ee84f7 (DEFAULT_TABLE table=test-table [id=207a41a359824b7d8c840d4df2b13db9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:59:48.870872 3856 tablet_service.cc:1467] Processing CreateTablet for tablet f796bbdd7f9e4e0d9c797cfaa8ee84f7 (DEFAULT_TABLE table=test-table [id=207a41a359824b7d8c840d4df2b13db9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 00:59:48.871204 3856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f796bbdd7f9e4e0d9c797cfaa8ee84f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:59:48.871204 3711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f796bbdd7f9e4e0d9c797cfaa8ee84f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 00:59:48.874094 4080 tablet_bootstrap.cc:492] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 556b8380958d45f9bcbf2aa0970f10a4: Bootstrap starting.
I20260806 00:59:48.875836 4080 tablet_bootstrap.cc:654] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 556b8380958d45f9bcbf2aa0970f10a4: Neither blocks nor log segments found. Creating new log.
I20260806 00:59:48.876286 4080 log.cc:824] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 556b8380958d45f9bcbf2aa0970f10a4: Log is configured to *not* fsync() on all Append() calls
I20260806 00:59:48.876808 4083 tablet_bootstrap.cc:492] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a: Bootstrap starting.
I20260806 00:59:48.879427 4083 tablet_bootstrap.cc:654] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a: Neither blocks nor log segments found. Creating new log.
I20260806 00:59:48.880007 4082 tablet_bootstrap.cc:492] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9cc759f9a7814c3791aab1de039bbcad: Bootstrap starting.
I20260806 00:59:48.881695 4082 tablet_bootstrap.cc:654] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9cc759f9a7814c3791aab1de039bbcad: Neither blocks nor log segments found. Creating new log.
I20260806 00:59:48.882382 4080 tablet_bootstrap.cc:492] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 556b8380958d45f9bcbf2aa0970f10a4: No bootstrap required, opened a new log
I20260806 00:59:48.882572 4080 ts_tablet_manager.cc:1397] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 556b8380958d45f9bcbf2aa0970f10a4: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260806 00:59:48.886181 4080 raft_consensus.cc:348] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 556b8380958d45f9bcbf2aa0970f10a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556b8380958d45f9bcbf2aa0970f10a4" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36391 } } peers { permanent_uuid: "9cc759f9a7814c3791aab1de039bbcad" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 37893 } } peers { permanent_uuid: "9423c0b9034d4e64a955cb94b9c9426a" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45021 } }
I20260806 00:59:48.886672 4080 raft_consensus.cc:374] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 556b8380958d45f9bcbf2aa0970f10a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:59:48.886802 4080 raft_consensus.cc:729] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 556b8380958d45f9bcbf2aa0970f10a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 556b8380958d45f9bcbf2aa0970f10a4, State: Initialized, Role: FOLLOWER
I20260806 00:59:48.887063 4080 consensus_queue.cc:260] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 556b8380958d45f9bcbf2aa0970f10a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556b8380958d45f9bcbf2aa0970f10a4" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36391 } } peers { permanent_uuid: "9cc759f9a7814c3791aab1de039bbcad" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 37893 } } peers { permanent_uuid: "9423c0b9034d4e64a955cb94b9c9426a" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45021 } }
I20260806 00:59:48.887586 4080 ts_tablet_manager.cc:1428] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 556b8380958d45f9bcbf2aa0970f10a4: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260806 00:59:48.887894 4059 heartbeater.cc:503] Master 127.23.242.62:44247 was elected leader, sending a full tablet report...
I20260806 00:59:48.888883 4082 log.cc:824] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9cc759f9a7814c3791aab1de039bbcad: Log is configured to *not* fsync() on all Append() calls
I20260806 00:59:48.888330 4083 log.cc:824] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a: Log is configured to *not* fsync() on all Append() calls
I20260806 00:59:48.890376 4083 tablet_bootstrap.cc:492] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a: No bootstrap required, opened a new log
I20260806 00:59:48.890396 4082 tablet_bootstrap.cc:492] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9cc759f9a7814c3791aab1de039bbcad: No bootstrap required, opened a new log
I20260806 00:59:48.890466 4082 ts_tablet_manager.cc:1397] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9cc759f9a7814c3791aab1de039bbcad: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260806 00:59:48.890466 4083 ts_tablet_manager.cc:1397] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20260806 00:59:48.893031 4083 raft_consensus.cc:348] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556b8380958d45f9bcbf2aa0970f10a4" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36391 } } peers { permanent_uuid: "9cc759f9a7814c3791aab1de039bbcad" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 37893 } } peers { permanent_uuid: "9423c0b9034d4e64a955cb94b9c9426a" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45021 } }
I20260806 00:59:48.893679 4083 raft_consensus.cc:374] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:59:48.893849 4083 raft_consensus.cc:729] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9423c0b9034d4e64a955cb94b9c9426a, State: Initialized, Role: FOLLOWER
I20260806 00:59:48.894165 4083 consensus_queue.cc:260] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556b8380958d45f9bcbf2aa0970f10a4" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36391 } } peers { permanent_uuid: "9cc759f9a7814c3791aab1de039bbcad" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 37893 } } peers { permanent_uuid: "9423c0b9034d4e64a955cb94b9c9426a" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45021 } }
I20260806 00:59:48.894573 4083 ts_tablet_manager.cc:1428] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260806 00:59:48.894774 4082 raft_consensus.cc:348] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9cc759f9a7814c3791aab1de039bbcad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556b8380958d45f9bcbf2aa0970f10a4" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36391 } } peers { permanent_uuid: "9cc759f9a7814c3791aab1de039bbcad" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 37893 } } peers { permanent_uuid: "9423c0b9034d4e64a955cb94b9c9426a" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45021 } }
I20260806 00:59:48.894937 3924 heartbeater.cc:503] Master 127.23.242.62:44247 was elected leader, sending a full tablet report...
I20260806 00:59:48.895077 4082 raft_consensus.cc:374] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9cc759f9a7814c3791aab1de039bbcad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 00:59:48.895176 4082 raft_consensus.cc:729] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9cc759f9a7814c3791aab1de039bbcad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cc759f9a7814c3791aab1de039bbcad, State: Initialized, Role: FOLLOWER
I20260806 00:59:48.896034 4082 consensus_queue.cc:260] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9cc759f9a7814c3791aab1de039bbcad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556b8380958d45f9bcbf2aa0970f10a4" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36391 } } peers { permanent_uuid: "9cc759f9a7814c3791aab1de039bbcad" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 37893 } } peers { permanent_uuid: "9423c0b9034d4e64a955cb94b9c9426a" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45021 } }
I20260806 00:59:48.896453 4082 ts_tablet_manager.cc:1428] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9cc759f9a7814c3791aab1de039bbcad: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 00:59:48.896634 3783 heartbeater.cc:503] Master 127.23.242.62:44247 was elected leader, sending a full tablet report...
W20260806 00:59:48.948935 3925 tablet.cc:2369] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 00:59:49.043365 4060 tablet.cc:2369] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 556b8380958d45f9bcbf2aa0970f10a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 00:59:49.134745 3784 tablet.cc:2369] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9cc759f9a7814c3791aab1de039bbcad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 00:59:49.140177 4088 raft_consensus.cc:482] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 00:59:49.140314 4088 raft_consensus.cc:504] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556b8380958d45f9bcbf2aa0970f10a4" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36391 } } peers { permanent_uuid: "9cc759f9a7814c3791aab1de039bbcad" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 37893 } } peers { permanent_uuid: "9423c0b9034d4e64a955cb94b9c9426a" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45021 } }
I20260806 00:59:49.140882 4088 leader_election.cc:288] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 556b8380958d45f9bcbf2aa0970f10a4 (127.23.242.3:36391), 9cc759f9a7814c3791aab1de039bbcad (127.23.242.1:37893)
I20260806 00:59:49.145123 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:59:49.141470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:36735 (local address 127.23.242.1:37893)
0806 00:59:49.141508 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0806 00:59:49.141509 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 00:59:49.141787 (+ 278us) server_negotiation.cc:407] Connection header received
0806 00:59:49.141890 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:59:49.141891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:59:49.141902 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:59:49.141930 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:59:49.142143 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:59:49.142882 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:59:49.144244 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:59:49.144751 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:59:49.144798 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:59:49.144808 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0806 00:59:49.145009 (+ 201us) server_negotiation.cc:299] Negotiation successful
0806 00:59:49.145046 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 00:59:49.145226 4075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:59:49.141383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:38487 (local address 127.23.242.3:36391)
0806 00:59:49.141428 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0806 00:59:49.141430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:59:49.141526 (+ 96us) server_negotiation.cc:407] Connection header received
0806 00:59:49.141614 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:59:49.141616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:59:49.141626 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:59:49.141692 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 00:59:49.141901 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:59:49.142744 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:59:49.144263 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:59:49.144806 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:59:49.144861 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:59:49.144870 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 00:59:49.145114 (+ 244us) server_negotiation.cc:299] Negotiation successful
0806 00:59:49.145157 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 00:59:49.145443 4091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:59:49.141107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:36391 (local address 127.23.242.2:38487)
0806 00:59:49.141453 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0806 00:59:49.141466 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 00:59:49.141538 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:59:49.141718 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:59:49.141719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:59:49.141721 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:59:49.141798 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:59:49.141801 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:59:49.142872 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:59:49.142873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:59:49.144145 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:59:49.144148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:59:49.144938 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:59:49.144939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:59:49.145019 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:59:49.145023 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 00:59:49.145312 (+ 289us) client_negotiation.cc:240] Negotiation successful
0806 00:59:49.145360 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":21,"thread_start_us":104,"threads_started":1}
I20260806 00:59:49.145915 4090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:59:49.141132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:37893 (local address 127.23.242.2:36735)
0806 00:59:49.141735 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0806 00:59:49.141746 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 00:59:49.141824 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 00:59:49.142000 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 00:59:49.142000 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 00:59:49.142002 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 00:59:49.142064 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:59:49.142066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:59:49.142985 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:59:49.142985 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:59:49.144124 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 00:59:49.144127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 00:59:49.144866 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 00:59:49.144866 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 00:59:49.144934 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:59:49.144938 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 00:59:49.145731 (+ 793us) client_negotiation.cc:240] Negotiation successful
0806 00:59:49.145776 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":2,"thread_start_us":80,"threads_started":1}
I20260806 00:59:49.146010 4001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f796bbdd7f9e4e0d9c797cfaa8ee84f7" candidate_uuid: "9423c0b9034d4e64a955cb94b9c9426a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "556b8380958d45f9bcbf2aa0970f10a4" is_pre_election: true
I20260806 00:59:49.146201 4001 raft_consensus.cc:2454] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 556b8380958d45f9bcbf2aa0970f10a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9423c0b9034d4e64a955cb94b9c9426a in term 0.
I20260806 00:59:49.146205 3735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f796bbdd7f9e4e0d9c797cfaa8ee84f7" candidate_uuid: "9423c0b9034d4e64a955cb94b9c9426a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cc759f9a7814c3791aab1de039bbcad" is_pre_election: true
I20260806 00:59:49.146343 3735 raft_consensus.cc:2454] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9cc759f9a7814c3791aab1de039bbcad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9423c0b9034d4e64a955cb94b9c9426a in term 0.
I20260806 00:59:49.146489 3812 leader_election.cc:302] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 556b8380958d45f9bcbf2aa0970f10a4, 9423c0b9034d4e64a955cb94b9c9426a; no voters:
I20260806 00:59:49.146714 4088 raft_consensus.cc:2781] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 00:59:49.146785 4088 raft_consensus.cc:482] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 00:59:49.146809 4088 raft_consensus.cc:3037] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:59:49.147771 4088 raft_consensus.cc:504] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556b8380958d45f9bcbf2aa0970f10a4" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36391 } } peers { permanent_uuid: "9cc759f9a7814c3791aab1de039bbcad" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 37893 } } peers { permanent_uuid: "9423c0b9034d4e64a955cb94b9c9426a" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45021 } }
I20260806 00:59:49.147917 4088 leader_election.cc:288] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a [CANDIDATE]: Term 1 election: Requested vote from peers 556b8380958d45f9bcbf2aa0970f10a4 (127.23.242.3:36391), 9cc759f9a7814c3791aab1de039bbcad (127.23.242.1:37893)
I20260806 00:59:49.148145 4001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f796bbdd7f9e4e0d9c797cfaa8ee84f7" candidate_uuid: "9423c0b9034d4e64a955cb94b9c9426a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "556b8380958d45f9bcbf2aa0970f10a4"
I20260806 00:59:49.148082 3735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f796bbdd7f9e4e0d9c797cfaa8ee84f7" candidate_uuid: "9423c0b9034d4e64a955cb94b9c9426a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cc759f9a7814c3791aab1de039bbcad"
I20260806 00:59:49.148226 4001 raft_consensus.cc:3037] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 556b8380958d45f9bcbf2aa0970f10a4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:59:49.148260 3735 raft_consensus.cc:3037] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9cc759f9a7814c3791aab1de039bbcad [term 0 FOLLOWER]: Advancing to term 1
I20260806 00:59:49.149159 4001 raft_consensus.cc:2454] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 556b8380958d45f9bcbf2aa0970f10a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9423c0b9034d4e64a955cb94b9c9426a in term 1.
I20260806 00:59:49.149242 3735 raft_consensus.cc:2454] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9cc759f9a7814c3791aab1de039bbcad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9423c0b9034d4e64a955cb94b9c9426a in term 1.
I20260806 00:59:49.149338 3812 leader_election.cc:302] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 556b8380958d45f9bcbf2aa0970f10a4, 9423c0b9034d4e64a955cb94b9c9426a; no voters:
I20260806 00:59:49.149471 4088 raft_consensus.cc:2781] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 00:59:49.149642 4088 raft_consensus.cc:686] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a [term 1 LEADER]: Becoming Leader. State: Replica: 9423c0b9034d4e64a955cb94b9c9426a, State: Running, Role: LEADER
I20260806 00:59:49.149753 4088 consensus_queue.cc:237] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556b8380958d45f9bcbf2aa0970f10a4" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36391 } } peers { permanent_uuid: "9cc759f9a7814c3791aab1de039bbcad" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 37893 } } peers { permanent_uuid: "9423c0b9034d4e64a955cb94b9c9426a" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45021 } }
I20260806 00:59:49.150619 3564 catalog_manager.cc:5495] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9423c0b9034d4e64a955cb94b9c9426a (127.23.242.2). New cstate: current_term: 1 leader_uuid: "9423c0b9034d4e64a955cb94b9c9426a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556b8380958d45f9bcbf2aa0970f10a4" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cc759f9a7814c3791aab1de039bbcad" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 37893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9423c0b9034d4e64a955cb94b9c9426a" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 45021 } health_report { overall_health: HEALTHY } } }
I20260806 00:59:49.210603 4079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:59:49.207202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56738 (local address 127.23.242.2:45021)
0806 00:59:49.207254 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0806 00:59:49.207256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 00:59:49.207294 (+ 38us) server_negotiation.cc:407] Connection header received
0806 00:59:49.207343 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 00:59:49.207344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 00:59:49.207364 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 00:59:49.207423 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 00:59:49.207636 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:59:49.208301 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:59:49.209514 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 00:59:49.209804 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 00:59:49.209886 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 00:59:49.210081 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 00:59:49.210242 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 00:59:49.210331 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 00:59:49.210450 (+ 119us) server_negotiation.cc:299] Negotiation successful
0806 00:59:49.210474 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 00:59:49.212836 4001 raft_consensus.cc:1261] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 556b8380958d45f9bcbf2aa0970f10a4 [term 1 FOLLOWER]: Refusing update from remote peer 9423c0b9034d4e64a955cb94b9c9426a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:59:49.212842 3735 raft_consensus.cc:1261] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9cc759f9a7814c3791aab1de039bbcad [term 1 FOLLOWER]: Refusing update from remote peer 9423c0b9034d4e64a955cb94b9c9426a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 00:59:49.213191 4088 consensus_queue.cc:1035] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a [LEADER]: Connected to new peer: Peer: permanent_uuid: "556b8380958d45f9bcbf2aa0970f10a4" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 00:59:49.213333 4092 consensus_queue.cc:1035] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cc759f9a7814c3791aab1de039bbcad" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 37893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Aug 06 00:59:49 dist-test-slave-dhph krb5kdc[3080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977988, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for HTTP/127.23.242.20@KRBTEST.COM
I20260806 00:59:49.223939 4096 mvcc.cc:205] Tried to move back new op lower bound from 7315365843812360192 to 7315365843558006784. Current Snapshot: MvccSnapshot[applied={T|T < 7315365843812360192 or (T in {7315365843812360192})}]
I20260806 00:59:49.225335 3564 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:59:49.225458 3564 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 00:59:49.226470 4097 mvcc.cc:205] Tried to move back new op lower bound from 7315365843812360192 to 7315365843558006784. Current Snapshot: MvccSnapshot[applied={T|T < 7315365843812360192}]
I20260806 00:59:49.227173 3564 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a86e83b51d95430791ea4834db586e31: Sending DeleteTablet for 3 replicas of tablet f796bbdd7f9e4e0d9c797cfaa8ee84f7
I20260806 00:59:49.227495 3993 tablet_service.cc:1514] Processing DeleteTablet for tablet f796bbdd7f9e4e0d9c797cfaa8ee84f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:59:49 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:57912
I20260806 00:59:49.227488 3711 tablet_service.cc:1514] Processing DeleteTablet for tablet f796bbdd7f9e4e0d9c797cfaa8ee84f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:59:49 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:48080
I20260806 00:59:49.227769 3855 tablet_service.cc:1514] Processing DeleteTablet for tablet f796bbdd7f9e4e0d9c797cfaa8ee84f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:59:49 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:56722
I20260806 00:59:49.227769 3856 tablet_service.cc:1514] Processing DeleteTablet for tablet f796bbdd7f9e4e0d9c797cfaa8ee84f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 00:59:49 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:56722
I20260806 00:59:49.227981 4098 mvcc.cc:205] Tried to move back new op lower bound from 7315365843812360192 to 7315365843558006784. Current Snapshot: MvccSnapshot[applied={T|T < 7315365843812360192}]
I20260806 00:59:49.228518 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 3650
I20260806 00:59:49.239861 4106 tablet_replica.cc:330] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 556b8380958d45f9bcbf2aa0970f10a4: stopping tablet replica
I20260806 00:59:49.240056 4106 raft_consensus.cc:2229] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 556b8380958d45f9bcbf2aa0970f10a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 00:59:49.240160 4106 raft_consensus.cc:2258] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 556b8380958d45f9bcbf2aa0970f10a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:59:49.240476 4107 tablet_replica.cc:330] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a: stopping tablet replica
I20260806 00:59:49.240661 4107 raft_consensus.cc:2229] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a [term 1 LEADER]: Raft consensus shutting down.
I20260806 00:59:49.240847 4107 raft_consensus.cc:2258] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 00:59:49.241484 4107 ts_tablet_manager.cc:1905] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:59:49.243463 4107 ts_tablet_manager.cc:1918] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 00:59:49.246621 4106 ts_tablet_manager.cc:1905] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 556b8380958d45f9bcbf2aa0970f10a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 00:59:49.250191 4106 ts_tablet_manager.cc:1918] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 556b8380958d45f9bcbf2aa0970f10a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 00:59:49.250435 4107 log.cc:1196] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-1/wal/wals/f796bbdd7f9e4e0d9c797cfaa8ee84f7
I20260806 00:59:49.255638 4106 log.cc:1196] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 556b8380958d45f9bcbf2aa0970f10a4: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785977869987768-24520-0/minicluster-data/ts-2/wal/wals/f796bbdd7f9e4e0d9c797cfaa8ee84f7
I20260806 00:59:49.256405 4107 ts_tablet_manager.cc:1939] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 9423c0b9034d4e64a955cb94b9c9426a: Deleting consensus metadata
I20260806 00:59:49.256873 4106 ts_tablet_manager.cc:1939] T f796bbdd7f9e4e0d9c797cfaa8ee84f7 P 556b8380958d45f9bcbf2aa0970f10a4: Deleting consensus metadata
I20260806 00:59:49.267580 3550 catalog_manager.cc:4841] TS 9423c0b9034d4e64a955cb94b9c9426a (127.23.242.2:45021): tablet f796bbdd7f9e4e0d9c797cfaa8ee84f7 (table test-table [id=207a41a359824b7d8c840d4df2b13db9]) successfully deleted
I20260806 00:59:49.268301 3552 catalog_manager.cc:4841] TS 556b8380958d45f9bcbf2aa0970f10a4 (127.23.242.3:36391): tablet f796bbdd7f9e4e0d9c797cfaa8ee84f7 (table test-table [id=207a41a359824b7d8c840d4df2b13db9]) successfully deleted
W20260806 00:59:49.268819 3550 catalog_manager.cc:4808] TS 9423c0b9034d4e64a955cb94b9c9426a (127.23.242.2:45021): delete failed for tablet f796bbdd7f9e4e0d9c797cfaa8ee84f7 because the tablet was not found. No further retry: Not found: Tablet not found: f796bbdd7f9e4e0d9c797cfaa8ee84f7
W20260806 00:59:49.272716 3549 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.242.1:37893 (error 108)
I20260806 00:59:49.275561 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 3789
I20260806 00:59:49.283164 4076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 00:59:49.282920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:37893 (local address 127.0.0.1:48090)
0806 00:59:49.283053 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 00:59:49.283113 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.1:37893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260806 00:59:49.283330 3549 catalog_manager.cc:4584] TS 9cc759f9a7814c3791aab1de039bbcad (127.23.242.1:37893): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f796bbdd7f9e4e0d9c797cfaa8ee84f7: Network error: Client connection negotiation failed: client connection to 127.23.242.1:37893: connect: Connection refused (error 111)
I20260806 00:59:49.288909 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 3929
I20260806 00:59:49.305944 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 3533
I20260806 00:59:49.316404 24520 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 00:59:49.672851 24520 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 00:59:49.673000 24520 mini_ranger.cc:66] Stopping Ranger...
I20260806 00:59:50.018802 24520 mini_ranger.cc:68] Stopped Ranger
2026-08-06 00:59:50.019 UTC [3110] LOG: received smart shutdown request
2026-08-06 00:59:50.023 UTC [3110] LOG: background worker "logical replication launcher" (PID 3119) exited with exit code 1
2026-08-06 00:59:50.024 UTC [3114] LOG: shutting down
2026-08-06 00:59:50.252 UTC [3110] LOG: database system is shut down
2026-08-06T00:59:50Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (58198 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 00:59:50 dist-test-slave-dhph krb5kdc[4119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 00:59:50 dist-test-slave-dhph krb5kdc[4119](info): set up 2 sockets
Aug 06 00:59:50 dist-test-slave-dhph krb5kdc[4119](info): commencing operation
krb5kdc: starting...
W20260806 00:59:51.558091 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 00:59:51 dist-test-slave-dhph krb5kdc[4119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785977991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T00:59:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T00:59:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.242.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.242.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.242.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/rangeradmin_127.23.242.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.242.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/rangeradmin_127.23.242.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.242.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.242.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.242.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/rangerlookup_127.23.242.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.242.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/rangerlookup_127.23.242.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.242.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.242.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.242.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/HTTP_127.23.242.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.242.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/HTTP_127.23.242.20@KRBTEST.COM.keytab.
I20260806 00:59:51.771169 24520 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskh8enal/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/postgres -l logfile start
2026-08-06 00:59:55.008 UTC [4150] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 00:59:55.008 UTC [4150] LOG: listening on IPv4 address "127.23.242.20", port 51935
2026-08-06 00:59:55.015 UTC [4150] LOG: listening on Unix socket "/tmp/.s.PGSQL.51935"
2026-08-06 00:59:55.041 UTC [4153] LOG: database system was shut down at 2026-08-06 00:59:52 UTC
2026-08-06 00:59:55.047 UTC [4150] LOG: database system is ready to accept connections
I20260806 00:59:56.104810 24520 mini_postgres.cc:96] Postgres bound to 51935
2026-08-06 00:59:56.112 UTC [4161] FATAL: database "slave" does not exist
127.23.242.20:51935 - accepting connections
I20260806 00:59:56.113214 24520 mini_ranger.cc:161] Starting Ranger...
I20260806 00:59:56.131138 24520 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 00:59:56.832764 24520 mini_ranger.cc:87] Created ranger Postgres database
I20260806 00:59:56.832875 24520 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ranger-admin
2026-08-06 00:59:57,152 [I] DB FLAVOR :POSTGRES
2026-08-06 00:59:57,153 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 00:59:57,153 [I] Checking connection..
2026-08-06 00:59:57,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:51935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 00:59:57,464 [I] Checking connection passed.
2026-08-06 00:59:57,465 [I] --------- Verifying version history table ---------
2026-08-06 00:59:57,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:51935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 00:59:57,753 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 00:59:57,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:51935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 00:59:58,034 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 00:59:58,034 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 00:59:58,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:51935/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 00:59:58.316 UTC [4227] WARNING: there is no transaction in progress
2026-08-06 00:59:58,324 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 00:59:58,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:51935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 00:59:58,616 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 00:59:58,616 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 00:59:58,616 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:51935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 00:59:58,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:51935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 00:59:59,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:51935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dhph.c.gcp-upstream.internal','N') ;"
2026-08-06 00:59:59,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:51935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 00:59:59,675 [I] Table x_portal_user does not exist in database ranger
2026-08-06 00:59:59,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:51935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 00:59:59,975 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 00:59:59,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:51935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 01:00:00,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:51935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 01:00:00,555 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 01:00:00,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:51935/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 01:00:01.875 UTC [4337] WARNING: there is no transaction in progress
2026-08-06 01:00:01.905 UTC [4337] WARNING: there is no transaction in progress
2026-08-06 01:00:01.936 UTC [4337] WARNING: there is no transaction in progress
2026-08-06 01:00:01.962 UTC [4337] WARNING: there is no transaction in progress
2026-08-06 01:00:01.992 UTC [4337] WARNING: there is no transaction in progress
2026-08-06 01:00:02.021 UTC [4337] WARNING: there is no transaction in progress
2026-08-06 01:00:02.114 UTC [4337] WARNING: there is no transaction in progress
2026-08-06 01:00:02.126 UTC [4337] WARNING: there is no transaction in progress
2026-08-06 01:00:02.150 UTC [4337] WARNING: there is no transaction in progress
2026-08-06 01:00:02.165 UTC [4337] WARNING: there is no transaction in progress
2026-08-06 01:00:02.180 UTC [4337] WARNING: there is no transaction in progress
2026-08-06 01:00:02.204 UTC [4337] WARNING: there is no transaction in progress
2026-08-06 01:00:02.219 UTC [4337] WARNING: there is no transaction in progress
2026-08-06 01:00:02.232 UTC [4337] WARNING: there is no transaction in progress
2026-08-06 01:00:02.245 UTC [4337] WARNING: there is no transaction in progress
2026-08-06 01:00:02.259 UTC [4337] WARNING: there is no transaction in progress
2026-08-06 01:00:03,491 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 01:00:03,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:51935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 01:00:03,795 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 01:00:03,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:51935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 01:00:04,155 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 01:00:04,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:51935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 01:00:04,497 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 01:00:04,497 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:51935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 01:00:04,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:51935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 01:00:05,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:51935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dhph.c.gcp-upstream.internal';"
2026-08-06 01:00:05,450 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 01:00:05,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:51935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 01:00:05,729 [I] DB_PATCHES have already been applied
I20260806 01:00:05.737282 24520 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ranger-admin:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskh8enal/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 01:00:05.737383 24520 mini_ranger.cc:193] Using host: 127.23.242.20
I20260806 01:00:05.743201 24520 mini_ranger.cc:227] Ranger admin URL: http://127.23.242.20:55199
Aug 06, 2026 1:00:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 1:00:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 1:00:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 1:00:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 1:00:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 1:00:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 1:00:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 01:00:07.047761 24520 mini_ranger.cc:160] Time spent starting Ranger: real 10.935s user 0.000s sys 0.015s
Aug 06, 2026 1:00:30 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 1:00:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 1:00:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 1:00:31 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 1:00:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 1:00:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 1:00:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 1:00:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 1:00:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 1:00:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 1:00:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 1:00:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 1:00:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 1:00:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 1:00:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 1:00:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 1:00:31 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 1:00:32 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 01:00:32.893343 24520 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.242.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.242.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.242.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/rangerkms_127.23.242.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.242.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/rangerkms_127.23.242.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.242.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.242.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.242.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/HTTP_127.23.242.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.242.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/HTTP_127.23.242.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 01:00:32 dist-test-slave-dhph krb5kdc[4119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978032, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 01:00:32.996515 24520 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 01:00:33.011457 24520 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 01:00:34.265172 24520 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 01:00:34.265300 24520 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ranger-kms
I20260806 01:00:34.265321 24520 mini_ranger_kms.cc:227] Using postgres at 127.23.242.20:51935
2026-08-06 01:00:34,462 [I] DB FLAVOR :POSTGRES
2026-08-06 01:00:34,463 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 01:00:34,463 [I] Checking connection
2026-08-06 01:00:34,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:51935/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 01:00:34,766 [I] connection success
2026-08-06 01:00:34,766 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 01:00:34,767 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 01:00:34,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:51935/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 01:00:35,060 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 01:00:35,060 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 01:00:35,061 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 01:00:35,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.242.20:51935/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 01:00:35,407 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 01:00:36.052716 24520 mini_ranger_kms.cc:310] Created kms service
I20260806 01:00:36.297600 24520 mini_ranger_kms.cc:326] Created kudu user
I20260806 01:00:36.357781 24520 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 01:00:36 dist-test-slave-dhph krb5kdc[4119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978032, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.242.20@KRBTEST.COM
I20260806 01:00:36.863831 24520 mini_ranger_kms.cc:384] Added ranger policy
I20260806 01:00:36.863968 24520 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ranger-kms:/tmp/dist-test-taskh8enal/build/release/bin/postgresql.jar:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskh8enal/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 01:00:36.863991 24520 mini_ranger_kms.cc:240] Using host: 127.23.242.20
I20260806 01:00:36.869791 24520 mini_ranger_kms.cc:276] Ranger KMS PID: 4519
I20260806 01:00:36.869921 24520 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.242.20:38073
Aug 06, 2026 1:00:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 1:00:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 1:00:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 1:00:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 1:00:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 1:00:38 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260806 01:00:39.383509 24520 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.387s user 0.000s sys 0.176s
I20260806 01:00:39.383816 24520 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 01:00:39.383906 24520 mini_ranger_kms.cc:401] 127.23.242.20:38073/kms/v1/keys
Aug 06 01:00:39 dist-test-slave-dhph krb5kdc[4119](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785978039, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.242.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 06 01:00:40 dist-test-slave-dhph krb5kdc[4119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978039, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.242.20@KRBTEST.COM for HTTP/127.23.242.20@KRBTEST.COM
Aug 06 01:00:40 dist-test-slave-dhph krb5kdc[4119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978039, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.242.20@KRBTEST.COM for HTTP/127.23.242.20@KRBTEST.COM
Aug 06 01:00:43 dist-test-slave-dhph krb5kdc[4119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 01:00:43.090842 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:44869
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:44869
--ranger_config_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.242.20:38073/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:00:43.205262 4571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:00:43.205509 4571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:00:43.205535 4571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:00:43.206910 4571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:00:43.206960 4571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:00:43.206975 4571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:00:43.206990 4571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:00:43.207003 4571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:00:43.208719 4571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.242.20:38073/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:44869
--ranger_config_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:44869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:00:43.208943 4571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:00:43.209129 4571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:00:43.215059 4579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:43.215075 4577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:43.215111 4576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:43.215440 4571 server_base.cc:1034] running on GCE node
I20260806 01:00:43.215782 4571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:00:43.216114 4571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:00:43.217298 4571 hybrid_clock.cc:648] HybridClock initialized: now 1785978043217264 us; error 64 us; skew 500 ppm
Aug 06 01:00:43 dist-test-slave-dhph krb5kdc[4119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978043, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:00:43.221051 4571 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 01:00:43.221814 4571 webserver.cc:460] Webserver started at http://127.23.242.62:33567/ using document root <none> and password file <none>
I20260806 01:00:43.222078 4571 fs_manager.cc:359] Metadata directory not provided
I20260806 01:00:43.222149 4571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:00:43.222316 4571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 01:00:43 dist-test-slave-dhph krb5kdc[4119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978043, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for HTTP/127.23.242.20@KRBTEST.COM
I20260806 01:00:43.293123 4571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "06df027bb53c4e11aba1e83695bb996c"
format_stamp: "Formatted at 2026-08-06 01:00:43 on dist-test-slave-dhph"
server_key: "073368a48bfe744d58bf1b2cb822e472"
server_key_iv: "fb4c1a08833f65e103654d13b4b29800"
server_key_version: "kuduclusterkey@0"
I20260806 01:00:43.293593 4571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "06df027bb53c4e11aba1e83695bb996c"
format_stamp: "Formatted at 2026-08-06 01:00:43 on dist-test-slave-dhph"
server_key: "073368a48bfe744d58bf1b2cb822e472"
server_key_iv: "fb4c1a08833f65e103654d13b4b29800"
server_key_version: "kuduclusterkey@0"
I20260806 01:00:43.295181 4571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260806 01:00:43.312309 4586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:43.312649 4571 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:00:43.312785 4571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "06df027bb53c4e11aba1e83695bb996c"
format_stamp: "Formatted at 2026-08-06 01:00:43 on dist-test-slave-dhph"
server_key: "073368a48bfe744d58bf1b2cb822e472"
server_key_iv: "fb4c1a08833f65e103654d13b4b29800"
server_key_version: "kuduclusterkey@0"
I20260806 01:00:43.312896 4571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:00:43.330323 4571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:00:43.332595 4571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:00:43.332728 4571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:00:43.337795 4571 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:44869
I20260806 01:00:43.337888 4638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:44869 every 8 connection(s)
I20260806 01:00:43.338261 4571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:00:43.339010 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:00:43.339754 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 4571
I20260806 01:00:43.339862 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:00:43.340077 24520 external_mini_cluster.cc:1428] Setting key 2d19428ea1d45e67729531069208ce58
I20260806 01:00:43.343035 4639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c: Bootstrap starting.
I20260806 01:00:43.343950 4639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c: Neither blocks nor log segments found. Creating new log.
I20260806 01:00:43.344249 4639 log.cc:824] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c: Log is configured to *not* fsync() on all Append() calls
I20260806 01:00:43.345008 4639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c: No bootstrap required, opened a new log
I20260806 01:00:43.348254 4639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06df027bb53c4e11aba1e83695bb996c" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44869 } }
I20260806 01:00:43.348455 4639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:00:43.348526 4639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06df027bb53c4e11aba1e83695bb996c, State: Initialized, Role: FOLLOWER
I20260806 01:00:43.348699 4639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06df027bb53c4e11aba1e83695bb996c" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44869 } }
I20260806 01:00:43.348805 4639 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:00:43.348863 4639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:00:43.348937 4639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [term 0 FOLLOWER]: Advancing to term 1
Aug 06 01:00:43 dist-test-slave-dhph krb5kdc[4119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:00:43.349643 4639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06df027bb53c4e11aba1e83695bb996c" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44869 } }
I20260806 01:00:43.349767 4639 leader_election.cc:302] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06df027bb53c4e11aba1e83695bb996c; no voters:
I20260806 01:00:43.350001 4639 leader_election.cc:288] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:00:43.350359 4639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:00:43.350037 4644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:00:43.350673 4644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [term 1 LEADER]: Becoming Leader. State: Replica: 06df027bb53c4e11aba1e83695bb996c, State: Running, Role: LEADER
I20260806 01:00:43.351270 4644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06df027bb53c4e11aba1e83695bb996c" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44869 } }
I20260806 01:00:43.352016 4645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06df027bb53c4e11aba1e83695bb996c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06df027bb53c4e11aba1e83695bb996c" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44869 } } }
I20260806 01:00:43.352104 4645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [sys.catalog]: This master's current role is: LEADER
I20260806 01:00:43.352334 4639 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 01:00:43.352440 4646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06df027bb53c4e11aba1e83695bb996c. Latest consensus state: current_term: 1 leader_uuid: "06df027bb53c4e11aba1e83695bb996c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06df027bb53c4e11aba1e83695bb996c" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44869 } } }
I20260806 01:00:43.352574 4646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [sys.catalog]: This master's current role is: LEADER
I20260806 01:00:43.352746 4658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:00:43.353296 4658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:00:43.357260 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:43.340678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41870 (local address 127.23.242.62:44869)
0806 01:00:43.340854 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:43.340864 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:43.340900 (+ 36us) server_negotiation.cc:407] Connection header received
0806 01:00:43.341676 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:43.341685 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:43.341764 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:43.341923 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:43.342775 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:43.344259 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:43.345362 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:43.346291 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:43.346516 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:43.349822 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:43.349827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:43.349828 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:43.349844 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:43.352071 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:43.352511 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:43.352514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:43.352514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:43.352611 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:43.352810 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:43.352811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:43.352811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:43.353130 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:43.353308 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:43.355173 (+ 1865us) server_negotiation.cc:299] Negotiation successful
0806 01:00:43.356579 (+ 1406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260806 01:00:43.363134 4658 catalog_manager.cc:1347] Generated new cluster ID: 3e320f9ef3784da5bf93b8ce8ccbac9c
I20260806 01:00:43.363217 4658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:00:43.387923 4658 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:00:43.388839 4658 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:00:43.396760 4658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c: Generated new TSK 0
I20260806 01:00:43.397074 4658 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.242.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
I20260806 01:00:44.163232 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.242.20:38073/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:0
--local_ip_for_outbound_sockets=127.23.242.1
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:44869
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:00:44.294849 4684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:00:44.295140 4684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:00:44.295184 4684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:00:44.296439 4684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:00:44.296511 4684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:00:44.296561 4684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 01:00:44.298192 4684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.242.20:38073/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:44869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:00:44.298492 4684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:00:44.298776 4684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:00:44.305444 4689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:44.305531 4690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:44.305907 4684 server_base.cc:1034] running on GCE node
W20260806 01:00:44.306939 4692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:44.307255 4684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:00:44.307595 4684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:00:44.308806 4684 hybrid_clock.cc:648] HybridClock initialized: now 1785978044308787 us; error 38 us; skew 500 ppm
Aug 06 01:00:44 dist-test-slave-dhph krb5kdc[4119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978044, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:00:44.312093 4684 init.cc:377] Logged in from keytab as kudu/127.23.242.1@KRBTEST.COM (short username kudu)
I20260806 01:00:44.312624 4684 webserver.cc:460] Webserver started at http://127.23.242.1:33519/ using document root <none> and password file <none>
I20260806 01:00:44.312883 4684 fs_manager.cc:359] Metadata directory not provided
I20260806 01:00:44.312954 4684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:00:44.313110 4684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 01:00:44 dist-test-slave-dhph krb5kdc[4119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978044, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for HTTP/127.23.242.20@KRBTEST.COM
I20260806 01:00:44.334537 4684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/data/instance:
uuid: "fc54ed87a2b44e699d59bf23361339fb"
format_stamp: "Formatted at 2026-08-06 01:00:44 on dist-test-slave-dhph"
server_key: "0ed2ab35d209e021e76d7593b5202248"
server_key_iv: "9684898731d95693cc79e4fce2256a00"
server_key_version: "kuduclusterkey@0"
I20260806 01:00:44.335122 4684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance:
uuid: "fc54ed87a2b44e699d59bf23361339fb"
format_stamp: "Formatted at 2026-08-06 01:00:44 on dist-test-slave-dhph"
server_key: "0ed2ab35d209e021e76d7593b5202248"
server_key_iv: "9684898731d95693cc79e4fce2256a00"
server_key_version: "kuduclusterkey@0"
I20260806 01:00:44.337579 4684 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 01:00:44.351806 4701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:44.352090 4684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:00:44.352188 4684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "fc54ed87a2b44e699d59bf23361339fb"
format_stamp: "Formatted at 2026-08-06 01:00:44 on dist-test-slave-dhph"
server_key: "0ed2ab35d209e021e76d7593b5202248"
server_key_iv: "9684898731d95693cc79e4fce2256a00"
server_key_version: "kuduclusterkey@0"
I20260806 01:00:44.352276 4684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:00:44.367715 4684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:00:44.370574 4684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:00:44.370870 4684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:00:44.371177 4684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:00:44.371665 4684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:00:44.371784 4684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:44.371902 4684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:00:44.371992 4684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:44.380698 4684 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:38717
I20260806 01:00:44.381584 4684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 01:00:44.384353 4817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:38717 every 8 connection(s)
I20260806 01:00:44.391762 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 4684
I20260806 01:00:44.391919 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance
I20260806 01:00:44.392153 24520 external_mini_cluster.cc:1428] Setting key 24f8811ff823ca0bcd475fb99f0a0862
Aug 06 01:00:44 dist-test-slave-dhph krb5kdc[4119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978044, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:00:44.399439 4820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:44.385374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:44869 (local address 127.23.242.1:52419)
0806 01:00:44.386053 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:44.386119 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:44.387025 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:44.387469 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:44.387474 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:44.387790 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:00:44.388110 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:44.388117 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:44.389360 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:44.389362 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:44.390710 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:44.390713 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:44.391296 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:44.391297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:44.391438 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:44.392110 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:00:44.392134 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:00:44.395293 (+ 3159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:00:44.397646 (+ 2353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:44.397655 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:44.397658 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:44.397988 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:44.398255 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:44.398256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:44.398257 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:44.398394 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:44.398723 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:00:44.398730 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:00:44.398900 (+ 170us) client_negotiation.cc:769] Sending connection context
0806 01:00:44.399002 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 01:00:44.399186 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":138,"threads_started":1}
I20260806 01:00:44.400151 4821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:44.385694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:52419 (local address 127.23.242.62:44869)
0806 01:00:44.386678 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:44.386681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:44.386694 (+ 13us) server_negotiation.cc:407] Connection header received
0806 01:00:44.387233 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:44.387235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:44.387250 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:44.387326 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:44.388236 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:44.389229 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:44.390813 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:44.391136 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:44.391205 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:44.395420 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:44.395422 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:44.395422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:44.395431 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:44.397536 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:44.398116 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:44.398117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:44.398118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:44.398187 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:44.398461 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:44.398461 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:44.398462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:44.398636 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:44.398696 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:44.399935 (+ 1239us) server_negotiation.cc:299] Negotiation successful
0806 01:00:44.400030 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":954,"thread_start_us":98,"threads_started":1}
I20260806 01:00:44.400916 4818 heartbeater.cc:347] Connected to a master server at 127.23.242.62:44869
I20260806 01:00:44.401013 4818 heartbeater.cc:464] Registering TS with master...
I20260806 01:00:44.401223 4818 heartbeater.cc:511] Master 127.23.242.62:44869 requested a full tablet report, sending...
I20260806 01:00:44.401788 4603 ts_manager.cc:194] Registered new tserver with Master: fc54ed87a2b44e699d59bf23361339fb (127.23.242.1:38717)
I20260806 01:00:44.402709 4603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.1@KRBTEST.COM'} at 127.23.242.1:52419
WARNING: no policy specified for kudu/127.23.242.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
I20260806 01:00:44.436349 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.242.20:38073/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:0
--local_ip_for_outbound_sockets=127.23.242.2
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:44869
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 01:00:44 dist-test-slave-dhph krb5kdc[4119](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785978044, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 01:00:44.581606 4824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:00:44.581945 4824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:00:44.581992 4824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:00:44.584484 4824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:00:44.584627 4824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:00:44.584702 4824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 01:00:44.587822 4824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.242.20:38073/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:44869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:00:44.588335 4824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:00:44.588704 4824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:00:44.597851 4833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:44.598470 4832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:44.599030 4824 server_base.cc:1034] running on GCE node
W20260806 01:00:44.599395 4835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:44.599864 4824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:00:44.600431 4824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:00:44.601653 4824 hybrid_clock.cc:648] HybridClock initialized: now 1785978044601627 us; error 74 us; skew 500 ppm
Aug 06 01:00:44 dist-test-slave-dhph krb5kdc[4119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978044, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:00:44.606066 4824 init.cc:377] Logged in from keytab as kudu/127.23.242.2@KRBTEST.COM (short username kudu)
I20260806 01:00:44.606762 4824 webserver.cc:460] Webserver started at http://127.23.242.2:41697/ using document root <none> and password file <none>
I20260806 01:00:44.607071 4824 fs_manager.cc:359] Metadata directory not provided
I20260806 01:00:44.607141 4824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:00:44.607278 4824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 01:00:44 dist-test-slave-dhph krb5kdc[4119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978044, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for HTTP/127.23.242.20@KRBTEST.COM
I20260806 01:00:44.627048 4824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/data/instance:
uuid: "8364a3ef9e2843d5a8ef71d5c885ec69"
format_stamp: "Formatted at 2026-08-06 01:00:44 on dist-test-slave-dhph"
server_key: "ab089a1410692c9ad50cede47b5d6377"
server_key_iv: "e97b8b0458bd2ca6cc106871978cde9b"
server_key_version: "kuduclusterkey@0"
I20260806 01:00:44.627931 4824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance:
uuid: "8364a3ef9e2843d5a8ef71d5c885ec69"
format_stamp: "Formatted at 2026-08-06 01:00:44 on dist-test-slave-dhph"
server_key: "ab089a1410692c9ad50cede47b5d6377"
server_key_iv: "e97b8b0458bd2ca6cc106871978cde9b"
server_key_version: "kuduclusterkey@0"
I20260806 01:00:44.630511 4824 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260806 01:00:44.646938 4842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:44.647444 4824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 01:00:44.647557 4824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "8364a3ef9e2843d5a8ef71d5c885ec69"
format_stamp: "Formatted at 2026-08-06 01:00:44 on dist-test-slave-dhph"
server_key: "ab089a1410692c9ad50cede47b5d6377"
server_key_iv: "e97b8b0458bd2ca6cc106871978cde9b"
server_key_version: "kuduclusterkey@0"
I20260806 01:00:44.647658 4824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:00:44.678043 4824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:00:44.683427 4824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:00:44.683645 4824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:00:44.683938 4824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:00:44.684378 4824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:00:44.684439 4824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:44.684482 4824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:00:44.684520 4824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:44.693147 4824 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:38863
I20260806 01:00:44.693691 4824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 01:00:44.694332 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 4824
I20260806 01:00:44.694484 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance
I20260806 01:00:44.694752 24520 external_mini_cluster.cc:1428] Setting key 8122b03e3a4306b0ff26c7ce5177495d
I20260806 01:00:44.702696 4957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:38863 every 8 connection(s)
Aug 06 01:00:44 dist-test-slave-dhph krb5kdc[4119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978044, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:00:44.709517 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:44.694642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:44869 (local address 127.23.242.2:48201)
0806 01:00:44.695265 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:44.695310 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:44.695951 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:44.698027 (+ 2076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:44.698032 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:44.698304 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:00:44.698666 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:44.698673 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:44.699735 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:44.699736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:44.701029 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:44.701033 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:44.701603 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:44.701605 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:44.701786 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:44.702475 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:00:44.702485 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:00:44.704743 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:00:44.707320 (+ 2577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:44.707332 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:44.707335 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:44.707733 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:44.708008 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:44.708009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:44.708009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:44.708196 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:44.708567 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:00:44.708574 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:00:44.708776 (+ 202us) client_negotiation.cc:769] Sending connection context
0806 01:00:44.708901 (+ 125us) client_negotiation.cc:240] Negotiation successful
0806 01:00:44.709100 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":103,"threads_started":1}
I20260806 01:00:44.710353 4821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:44.695079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:48201 (local address 127.23.242.62:44869)
0806 01:00:44.695199 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:44.695202 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:44.695462 (+ 260us) server_negotiation.cc:407] Connection header received
0806 01:00:44.697815 (+ 2353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:44.697817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:44.697838 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:44.697923 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:44.698778 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:44.699611 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:44.701141 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:44.701476 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:44.701584 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:44.704878 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:44.704881 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:44.704881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:44.704892 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:44.707197 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:44.707855 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:44.707857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:44.707858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:44.707932 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:44.708289 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:44.708290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:44.708291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:44.708461 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:44.708534 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:44.710143 (+ 1609us) server_negotiation.cc:299] Negotiation successful
0806 01:00:44.710255 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 01:00:44.711031 4958 heartbeater.cc:347] Connected to a master server at 127.23.242.62:44869
I20260806 01:00:44.711153 4958 heartbeater.cc:464] Registering TS with master...
I20260806 01:00:44.711377 4958 heartbeater.cc:511] Master 127.23.242.62:44869 requested a full tablet report, sending...
I20260806 01:00:44.711787 4603 ts_manager.cc:194] Registered new tserver with Master: 8364a3ef9e2843d5a8ef71d5c885ec69 (127.23.242.2:38863)
I20260806 01:00:44.712302 4603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.2@KRBTEST.COM'} at 127.23.242.2:48201
WARNING: no policy specified for kudu/127.23.242.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
I20260806 01:00:44.753485 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.242.20:38073/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:0
--local_ip_for_outbound_sockets=127.23.242.3
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:44869
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:00:44.901144 4964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:00:44.901475 4964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:00:44.901520 4964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:00:44.903582 4964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:00:44.903664 4964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:00:44.903710 4964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 01:00:44.906512 4964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.242.20:38073/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:44869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:00:44.906976 4964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:00:44.907310 4964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:00:44.915102 4972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:44.915104 4970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:44.915253 4969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:44.916067 4964 server_base.cc:1034] running on GCE node
I20260806 01:00:44.916419 4964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:00:44.916760 4964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:00:44.917946 4964 hybrid_clock.cc:648] HybridClock initialized: now 1785978044917913 us; error 46 us; skew 500 ppm
Aug 06 01:00:44 dist-test-slave-dhph krb5kdc[4119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978044, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:00:44.922560 4964 init.cc:377] Logged in from keytab as kudu/127.23.242.3@KRBTEST.COM (short username kudu)
I20260806 01:00:44.923163 4964 webserver.cc:460] Webserver started at http://127.23.242.3:40639/ using document root <none> and password file <none>
I20260806 01:00:44.923425 4964 fs_manager.cc:359] Metadata directory not provided
I20260806 01:00:44.923487 4964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:00:44.923622 4964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 01:00:44 dist-test-slave-dhph krb5kdc[4119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978044, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for HTTP/127.23.242.20@KRBTEST.COM
I20260806 01:00:44.940407 4964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/data/instance:
uuid: "8ff48d3b54894bedb5fab5f076a6230a"
format_stamp: "Formatted at 2026-08-06 01:00:44 on dist-test-slave-dhph"
server_key: "2c759857ab53b5b284959dca80820c11"
server_key_iv: "f7493f93c5b2374872b765da985b954a"
server_key_version: "kuduclusterkey@0"
I20260806 01:00:44.941011 4964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance:
uuid: "8ff48d3b54894bedb5fab5f076a6230a"
format_stamp: "Formatted at 2026-08-06 01:00:44 on dist-test-slave-dhph"
server_key: "2c759857ab53b5b284959dca80820c11"
server_key_iv: "f7493f93c5b2374872b765da985b954a"
server_key_version: "kuduclusterkey@0"
I20260806 01:00:44.944113 4964 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260806 01:00:44.970104 4980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:44.970489 4964 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 01:00:44.970611 4964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "8ff48d3b54894bedb5fab5f076a6230a"
format_stamp: "Formatted at 2026-08-06 01:00:44 on dist-test-slave-dhph"
server_key: "2c759857ab53b5b284959dca80820c11"
server_key_iv: "f7493f93c5b2374872b765da985b954a"
server_key_version: "kuduclusterkey@0"
I20260806 01:00:44.970708 4964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:00:44.985379 4964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:00:44.988417 4964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:00:44.988665 4964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:00:44.988952 4964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:00:44.989398 4964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:00:44.989486 4964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:44.989565 4964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:00:44.989629 4964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:44.997131 4964 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:46191
I20260806 01:00:44.997655 4964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
I20260806 01:00:45.000537 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 4964
I20260806 01:00:45.000661 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance
I20260806 01:00:45.000876 24520 external_mini_cluster.cc:1428] Setting key 065fb27d81799f98aebfb7e0aaa8263b
I20260806 01:00:45.002538 5096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:46191 every 8 connection(s)
Aug 06 01:00:45 dist-test-slave-dhph krb5kdc[4119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978044, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:00:45.016916 4821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:45.003638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:46185 (local address 127.23.242.62:44869)
0806 01:00:45.003703 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:45.003705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:45.004455 (+ 750us) server_negotiation.cc:407] Connection header received
0806 01:00:45.005336 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:45.005337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:45.005351 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:45.005384 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:45.006363 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:45.007316 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:45.008957 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:45.009530 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:45.009588 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:45.013363 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:45.013364 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:45.013365 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:45.013373 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:45.015147 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:45.015692 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:45.015693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:45.015693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:45.015754 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:45.016078 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:45.016079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:45.016079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:45.016234 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:45.016277 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:45.016714 (+ 437us) server_negotiation.cc:299] Negotiation successful
0806 01:00:45.016810 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 01:00:45.017747 5099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:45.003259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:44869 (local address 127.23.242.3:46185)
0806 01:00:45.004314 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:45.004365 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:45.005200 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:45.005613 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:45.005618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:45.005906 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:00:45.006220 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:45.006226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:45.007449 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:45.007450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:45.008829 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:45.008833 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:45.009755 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:45.009756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:45.009913 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:45.010749 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:00:45.010761 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:00:45.013244 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:00:45.015260 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:45.015271 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:45.015274 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:45.015600 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:45.015825 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:45.015825 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:45.015826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:45.016006 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:45.016325 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:00:45.016331 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:00:45.016504 (+ 173us) client_negotiation.cc:769] Sending connection context
0806 01:00:45.017319 (+ 815us) client_negotiation.cc:240] Negotiation successful
0806 01:00:45.017499 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1008,"thread_start_us":139,"threads_started":1}
I20260806 01:00:45.018407 5097 heartbeater.cc:347] Connected to a master server at 127.23.242.62:44869
I20260806 01:00:45.018504 5097 heartbeater.cc:464] Registering TS with master...
I20260806 01:00:45.019313 5097 heartbeater.cc:511] Master 127.23.242.62:44869 requested a full tablet report, sending...
I20260806 01:00:45.019953 4603 ts_manager.cc:194] Registered new tserver with Master: 8ff48d3b54894bedb5fab5f076a6230a (127.23.242.3:46191)
I20260806 01:00:45.020802 4603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.3@KRBTEST.COM'} at 127.23.242.3:46185
I20260806 01:00:45.023814 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 01:00:45.024760 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 4684
I20260806 01:00:45.038513 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 4824
I20260806 01:00:45.054348 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 4964
I20260806 01:00:45.062498 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 4571
I20260806 01:00:45.073028 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:44869
--webserver_interface=127.23.242.62
--webserver_port=33567
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:44869
--ranger_config_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.242.20:38073/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.242.20:38073/kms with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:00:45.178757 5100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:00:45.179044 5100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:00:45.179090 5100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:00:45.181084 5100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:00:45.181164 5100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:00:45.181226 5100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:00:45.181249 5100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:00:45.181279 5100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:00:45.184172 5100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.242.20:38073/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:44869
--ranger_config_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:44869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=33567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:00:45.184557 5100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:00:45.184875 5100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:00:45.191206 5106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:45.191375 5108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:45.191453 5105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:45.191421 5100 server_base.cc:1034] running on GCE node
I20260806 01:00:45.191744 5100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:00:45.192037 5100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:00:45.193256 5100 hybrid_clock.cc:648] HybridClock initialized: now 1785978045193220 us; error 54 us; skew 500 ppm
Aug 06 01:00:45 dist-test-slave-dhph krb5kdc[4119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978045, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:00:45.196566 5100 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 01:00:45.197365 5100 webserver.cc:460] Webserver started at http://127.23.242.62:33567/ using document root <none> and password file <none>
I20260806 01:00:45.197649 5100 fs_manager.cc:359] Metadata directory not provided
I20260806 01:00:45.197706 5100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 01:00:45 dist-test-slave-dhph krb5kdc[4119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978045, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for HTTP/127.23.242.20@KRBTEST.COM
I20260806 01:00:45.239126 5100 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 01:00:45.240594 5116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:45.240932 5100 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 01:00:45.241068 5100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "06df027bb53c4e11aba1e83695bb996c"
format_stamp: "Formatted at 2026-08-06 01:00:43 on dist-test-slave-dhph"
server_key: "073368a48bfe744d58bf1b2cb822e472"
server_key_iv: "fb4c1a08833f65e103654d13b4b29800"
server_key_version: "kuduclusterkey@0"
I20260806 01:00:45.241513 5100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:00:45.256943 5100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:00:45.259554 5100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:00:45.260022 5100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:00:45.266748 5100 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:44869
I20260806 01:00:45.266943 5169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:44869 every 8 connection(s)
I20260806 01:00:45.267341 5100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:00:45.268847 5170 sys_catalog.cc:263] Verifying existing consensus state
I20260806 01:00:45.269584 5170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c: Bootstrap starting.
I20260806 01:00:45.270769 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 5100
I20260806 01:00:45.271144 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.242.20:38073/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:38717
--local_ip_for_outbound_sockets=127.23.242.1
--tserver_master_addrs=127.23.242.62:44869
--webserver_port=33519
--webserver_interface=127.23.242.1
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.242.20:38073/kms with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 01:00:45.279558 5170 log.cc:824] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c: Log is configured to *not* fsync() on all Append() calls
I20260806 01:00:45.281718 5170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 01:00:45.282099 5170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c: Bootstrap complete.
I20260806 01:00:45.284950 5170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06df027bb53c4e11aba1e83695bb996c" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44869 } }
I20260806 01:00:45.285216 5170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06df027bb53c4e11aba1e83695bb996c, State: Initialized, Role: FOLLOWER
I20260806 01:00:45.285404 5170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06df027bb53c4e11aba1e83695bb996c" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44869 } }
I20260806 01:00:45.285487 5170 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:00:45.285547 5170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:00:45.285601 5170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [term 1 FOLLOWER]: Advancing to term 2
I20260806 01:00:45.286978 5170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06df027bb53c4e11aba1e83695bb996c" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44869 } }
I20260806 01:00:45.287078 5170 leader_election.cc:302] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06df027bb53c4e11aba1e83695bb996c; no voters:
I20260806 01:00:45.287295 5170 leader_election.cc:288] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 01:00:45.287375 5174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 01:00:45.287599 5170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:00:45.287631 5174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [term 2 LEADER]: Becoming Leader. State: Replica: 06df027bb53c4e11aba1e83695bb996c, State: Running, Role: LEADER
I20260806 01:00:45.288139 5174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06df027bb53c4e11aba1e83695bb996c" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44869 } }
I20260806 01:00:45.288976 5175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "06df027bb53c4e11aba1e83695bb996c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06df027bb53c4e11aba1e83695bb996c" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44869 } } }
I20260806 01:00:45.289062 5175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [sys.catalog]: This master's current role is: LEADER
I20260806 01:00:45.289011 5176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06df027bb53c4e11aba1e83695bb996c. Latest consensus state: current_term: 2 leader_uuid: "06df027bb53c4e11aba1e83695bb996c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06df027bb53c4e11aba1e83695bb996c" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44869 } } }
I20260806 01:00:45.289108 5176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c [sys.catalog]: This master's current role is: LEADER
I20260806 01:00:45.289326 5188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:00:45.289512 5170 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 01:00:45.290220 5188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:00:45.293229 5188 catalog_manager.cc:1259] Loaded cluster ID: 3e320f9ef3784da5bf93b8ce8ccbac9c
I20260806 01:00:45.293428 5188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:00:45.296046 5188 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:00:45.296304 5188 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c: Loaded TSK: 0
I20260806 01:00:45.296573 5188 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 01:00:45.409559 5172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:00:45.409991 5172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:00:45.410135 5172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:00:45.413375 5172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:00:45.413468 5172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:00:45.413522 5172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 01:00:45.416307 5172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.242.20:38073/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:38717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=33519
--tserver_master_addrs=127.23.242.62:44869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:00:45.416605 5172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:00:45.416923 5172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:00:45.425020 5210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:45.425341 5207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:45.425457 5172 server_base.cc:1034] running on GCE node
W20260806 01:00:45.426362 5208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:45.426708 5172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:00:45.427057 5172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:00:45.428306 5172 hybrid_clock.cc:648] HybridClock initialized: now 1785978045428280 us; error 44 us; skew 500 ppm
Aug 06 01:00:45 dist-test-slave-dhph krb5kdc[4119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978045, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:00:45.432097 5172 init.cc:377] Logged in from keytab as kudu/127.23.242.1@KRBTEST.COM (short username kudu)
I20260806 01:00:45.432754 5172 webserver.cc:460] Webserver started at http://127.23.242.1:33519/ using document root <none> and password file <none>
I20260806 01:00:45.433068 5172 fs_manager.cc:359] Metadata directory not provided
I20260806 01:00:45.433167 5172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 01:00:45 dist-test-slave-dhph krb5kdc[4119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978045, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for HTTP/127.23.242.20@KRBTEST.COM
I20260806 01:00:45.467348 5172 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.001s sys 0.000s
I20260806 01:00:45.469317 5219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:45.469550 5172 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:00:45.469673 5172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "fc54ed87a2b44e699d59bf23361339fb"
format_stamp: "Formatted at 2026-08-06 01:00:44 on dist-test-slave-dhph"
server_key: "0ed2ab35d209e021e76d7593b5202248"
server_key_iv: "9684898731d95693cc79e4fce2256a00"
server_key_version: "kuduclusterkey@0"
I20260806 01:00:45.470252 5172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:00:45.494488 5172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:00:45.497231 5172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:00:45.497421 5172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:00:45.497728 5172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:00:45.498139 5172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:00:45.498195 5172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:45.498236 5172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:00:45.498261 5172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:45.511528 5172 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:38717
I20260806 01:00:45.512059 5172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 01:00:45.512929 5333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:38717 every 8 connection(s)
I20260806 01:00:45.522269 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 5172
I20260806 01:00:45.522876 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.242.20:38073/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:38863
--local_ip_for_outbound_sockets=127.23.242.2
--tserver_master_addrs=127.23.242.62:44869
--webserver_port=41697
--webserver_interface=127.23.242.2
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.242.20:38073/kms with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 01:00:45 dist-test-slave-dhph krb5kdc[4119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978045, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:00:45.530005 5336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:45.513640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:44869 (local address 127.23.242.1:48539)
0806 01:00:45.514074 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:45.514117 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:45.514807 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:45.515610 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:45.515615 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:45.515876 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:00:45.516183 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:45.516189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:45.517955 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:45.517957 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:45.519507 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:45.519510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:45.520484 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:45.520486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:45.520629 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:45.522014 (+ 1385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:00:45.522025 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:00:45.523699 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:00:45.527495 (+ 3796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:45.527507 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:45.527510 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:45.527908 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:45.528314 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:45.528315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:45.528315 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:45.528474 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:45.529263 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:00:45.529270 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:00:45.529435 (+ 165us) client_negotiation.cc:769] Sending connection context
0806 01:00:45.529537 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 01:00:45.529742 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":98,"threads_started":1}
I20260806 01:00:45.531105 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:45.513733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:48539 (local address 127.23.242.62:44869)
0806 01:00:45.515055 (+ 1322us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:45.515065 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:45.515100 (+ 35us) server_negotiation.cc:407] Connection header received
0806 01:00:45.515263 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:45.515269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:45.515326 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:45.515457 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:45.516321 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:45.517824 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:45.519650 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:45.520344 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:45.520989 (+ 645us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:45.523861 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:45.523866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:45.523867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:45.523880 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:45.527330 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:45.528061 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:45.528065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:45.528065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:45.528205 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:45.528690 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:45.528691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:45.528691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:45.529157 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:45.530677 (+ 1520us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:45.530760 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 01:00:45.530898 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1258,"thread_start_us":111,"threads_started":1}
I20260806 01:00:45.532104 5334 heartbeater.cc:347] Connected to a master server at 127.23.242.62:44869
I20260806 01:00:45.532222 5334 heartbeater.cc:464] Registering TS with master...
I20260806 01:00:45.532426 5334 heartbeater.cc:511] Master 127.23.242.62:44869 requested a full tablet report, sending...
W20260806 01:00:45.533414 5334 heartbeater.cc:644] Failed to heartbeat to 127.23.242.62:44869 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 01:00:45.533492 5334 heartbeater.cc:464] Registering TS with master...
I20260806 01:00:45.533560 5334 heartbeater.cc:511] Master 127.23.242.62:44869 requested a full tablet report, sending...
I20260806 01:00:45.534011 5334 heartbeater.cc:464] Registering TS with master...
I20260806 01:00:45.534092 5334 heartbeater.cc:511] Master 127.23.242.62:44869 requested a full tablet report, sending...
W20260806 01:00:45.534466 5334 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 01:00:45.667815 5338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:00:45.668093 5338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:00:45.668139 5338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:00:45.670008 5338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:00:45.670100 5338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:00:45.670173 5338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 01:00:45.672068 5338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.242.20:38073/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:38863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=41697
--tserver_master_addrs=127.23.242.62:44869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:00:45.672433 5338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:00:45.672713 5338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:00:45.680310 5347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:45.680341 5345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:45.682693 5338 server_base.cc:1034] running on GCE node
W20260806 01:00:45.684700 5344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:45.685071 5338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:00:45.685444 5338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:00:45.686676 5338 hybrid_clock.cc:648] HybridClock initialized: now 1785978045686655 us; error 37 us; skew 500 ppm
Aug 06 01:00:45 dist-test-slave-dhph krb5kdc[4119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978045, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:00:45.691517 5338 init.cc:377] Logged in from keytab as kudu/127.23.242.2@KRBTEST.COM (short username kudu)
I20260806 01:00:45.692276 5338 webserver.cc:460] Webserver started at http://127.23.242.2:41697/ using document root <none> and password file <none>
I20260806 01:00:45.692636 5338 fs_manager.cc:359] Metadata directory not provided
I20260806 01:00:45.692741 5338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 01:00:45 dist-test-slave-dhph krb5kdc[4119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978045, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for HTTP/127.23.242.20@KRBTEST.COM
I20260806 01:00:45.716472 5338 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:00:45.719169 5355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:45.719468 5338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260806 01:00:45.719568 5338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "8364a3ef9e2843d5a8ef71d5c885ec69"
format_stamp: "Formatted at 2026-08-06 01:00:44 on dist-test-slave-dhph"
server_key: "ab089a1410692c9ad50cede47b5d6377"
server_key_iv: "e97b8b0458bd2ca6cc106871978cde9b"
server_key_version: "kuduclusterkey@0"
I20260806 01:00:45.720118 5338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:00:45.741636 5338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:00:45.746174 5338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:00:45.746480 5338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:00:45.746865 5338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:00:45.747344 5338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:00:45.747404 5338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:45.747460 5338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:00:45.747494 5338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:45.756217 5338 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:38863
I20260806 01:00:45.756286 5471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:38863 every 8 connection(s)
I20260806 01:00:45.756686 5338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
Aug 06 01:00:45 dist-test-slave-dhph krb5kdc[4119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978045, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:00:45.766114 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 5338
I20260806 01:00:45.766522 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.242.20:38073/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:46191
--local_ip_for_outbound_sockets=127.23.242.3
--tserver_master_addrs=127.23.242.62:44869
--webserver_port=40639
--webserver_interface=127.23.242.3
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:45753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.242.20:38073/kms with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 01:00:45.768810 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:45.757596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:57595 (local address 127.23.242.62:44869)
0806 01:00:45.757683 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:45.757686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:45.757927 (+ 241us) server_negotiation.cc:407] Connection header received
0806 01:00:45.758484 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:45.758486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:45.758505 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:45.758641 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:45.759863 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:45.760432 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:45.761677 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:45.761920 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:45.762016 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:45.764922 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:45.764924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:45.764924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:45.764933 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:45.766503 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:45.767151 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:45.767152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:45.767153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:45.767216 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:45.767487 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:45.767488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:45.767488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:45.767747 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:45.767878 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:45.768584 (+ 706us) server_negotiation.cc:299] Negotiation successful
0806 01:00:45.768695 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 01:00:45.769155 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:45.757573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:44869 (local address 127.23.242.2:57595)
0806 01:00:45.757761 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:45.757807 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:45.758346 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:45.758951 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:45.758955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:45.759171 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:00:45.759489 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:45.759713 (+ 224us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:45.760550 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:45.760552 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:45.761561 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:45.761571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:45.762046 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:45.762048 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:45.762207 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:45.762922 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:00:45.762932 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:00:45.764807 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:00:45.766757 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:45.766764 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:45.766767 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:45.767063 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:45.767289 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:45.767290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:45.767290 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:45.767411 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:45.767866 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:00:45.767873 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:00:45.768051 (+ 178us) client_negotiation.cc:769] Sending connection context
0806 01:00:45.768538 (+ 487us) client_negotiation.cc:240] Negotiation successful
0806 01:00:45.768751 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":116,"threads_started":1}
I20260806 01:00:45.770884 5472 heartbeater.cc:347] Connected to a master server at 127.23.242.62:44869
I20260806 01:00:45.771062 5472 heartbeater.cc:464] Registering TS with master...
I20260806 01:00:45.771319 5472 heartbeater.cc:511] Master 127.23.242.62:44869 requested a full tablet report, sending...
W20260806 01:00:45.772228 5472 heartbeater.cc:644] Failed to heartbeat to 127.23.242.62:44869 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 01:00:45.772318 5472 heartbeater.cc:464] Registering TS with master...
I20260806 01:00:45.772527 5472 heartbeater.cc:511] Master 127.23.242.62:44869 requested a full tablet report, sending...
I20260806 01:00:45.773217 5472 heartbeater.cc:464] Registering TS with master...
I20260806 01:00:45.773315 5472 heartbeater.cc:511] Master 127.23.242.62:44869 requested a full tablet report, sending...
W20260806 01:00:45.773758 5472 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 01:00:45.922456 5475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:00:45.922820 5475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:00:45.922863 5475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:00:45.925585 5475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:00:45.925699 5475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:00:45.925753 5475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 01:00:45.930379 5475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:45753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.242.20:38073/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:46191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=40639
--tserver_master_addrs=127.23.242.62:44869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:00:45.930872 5475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:00:45.931218 5475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:00:45.939358 5484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:45.939721 5481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:45.939918 5482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:45.940444 5475 server_base.cc:1034] running on GCE node
I20260806 01:00:45.940744 5475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:00:45.941196 5475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:00:45.942437 5475 hybrid_clock.cc:648] HybridClock initialized: now 1785978045942374 us; error 90 us; skew 500 ppm
Aug 06 01:00:45 dist-test-slave-dhph krb5kdc[4119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978045, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:00:45.946985 5475 init.cc:377] Logged in from keytab as kudu/127.23.242.3@KRBTEST.COM (short username kudu)
I20260806 01:00:45.948894 5475 webserver.cc:460] Webserver started at http://127.23.242.3:40639/ using document root <none> and password file <none>
I20260806 01:00:45.949373 5475 fs_manager.cc:359] Metadata directory not provided
I20260806 01:00:45.949476 5475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 01:00:45 dist-test-slave-dhph krb5kdc[4119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978045, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for HTTP/127.23.242.20@KRBTEST.COM
I20260806 01:00:45.975415 5475 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:00:45.979224 5492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:45.979519 5475 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260806 01:00:45.979655 5475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "8ff48d3b54894bedb5fab5f076a6230a"
format_stamp: "Formatted at 2026-08-06 01:00:44 on dist-test-slave-dhph"
server_key: "2c759857ab53b5b284959dca80820c11"
server_key_iv: "f7493f93c5b2374872b765da985b954a"
server_key_version: "kuduclusterkey@0"
I20260806 01:00:45.980362 5475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:00:46.001214 5475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:00:46.005226 5475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:00:46.005479 5475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:00:46.005775 5475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:00:46.006306 5475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:00:46.006382 5475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:46.006423 5475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:00:46.006445 5475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:46.016403 5475 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:46191
I20260806 01:00:46.017997 5475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
I20260806 01:00:46.022867 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 5475
I20260806 01:00:46.019250 5609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:46191 every 8 connection(s)
Aug 06 01:00:46 dist-test-slave-dhph krb5kdc[4119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978045, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:00:46.037955 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:46.019891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:53867 (local address 127.23.242.62:44869)
0806 01:00:46.023213 (+ 3322us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:46.023216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:46.023233 (+ 17us) server_negotiation.cc:407] Connection header received
0806 01:00:46.023266 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:46.023267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:46.023280 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:46.023375 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:46.025669 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:46.026724 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:46.028869 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:46.029447 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:46.029515 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:46.032907 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:46.032908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:46.032908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:46.032916 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:46.035257 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:46.036316 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:46.036317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:46.036318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:46.036387 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:46.036814 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:46.036815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:46.036815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:46.037012 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:46.037077 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:46.037711 (+ 634us) server_negotiation.cc:299] Negotiation successful
0806 01:00:46.037842 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3280}
I20260806 01:00:46.038156 5612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:46.019262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:44869 (local address 127.23.242.3:53867)
0806 01:00:46.019853 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:46.019903 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:46.020659 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:46.023642 (+ 2983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:46.023649 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:46.024666 (+ 1017us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:00:46.025393 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:46.025526 (+ 133us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:46.026893 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:46.026895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:46.028713 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:46.028721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:46.029699 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:46.029701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:46.029923 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:46.030896 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:00:46.030910 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:00:46.032785 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:00:46.035437 (+ 2652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:46.035450 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:46.035453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:46.036190 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:46.036473 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:46.036474 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:46.036474 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:46.036722 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:46.037282 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:00:46.037292 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:00:46.037502 (+ 210us) client_negotiation.cc:769] Sending connection context
0806 01:00:46.037649 (+ 147us) client_negotiation.cc:240] Negotiation successful
0806 01:00:46.037857 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":153,"threads_started":1}
I20260806 01:00:46.038880 5610 heartbeater.cc:347] Connected to a master server at 127.23.242.62:44869
I20260806 01:00:46.039247 5610 heartbeater.cc:464] Registering TS with master...
I20260806 01:00:46.039573 5610 heartbeater.cc:511] Master 127.23.242.62:44869 requested a full tablet report, sending...
W20260806 01:00:46.040572 5610 heartbeater.cc:644] Failed to heartbeat to 127.23.242.62:44869 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 01:00:46.040689 5610 heartbeater.cc:464] Registering TS with master...
I20260806 01:00:46.040766 5610 heartbeater.cc:511] Master 127.23.242.62:44869 requested a full tablet report, sending...
I20260806 01:00:46.040939 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:46.023862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41872 (local address 127.23.242.62:44869)
0806 01:00:46.024863 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:46.024865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:46.024874 (+ 9us) server_negotiation.cc:407] Connection header received
0806 01:00:46.024903 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:46.024903 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:46.024917 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:46.024986 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:46.026847 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:46.027965 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:46.029565 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:46.029884 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:46.029954 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:46.032832 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:46.032835 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:46.032836 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:46.032844 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:46.036561 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:46.037105 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:46.037106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:46.037106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:46.037178 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:46.038295 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:46.038296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:46.038297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:46.038450 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:46.038685 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:46.040109 (+ 1424us) server_negotiation.cc:299] Negotiation successful
0806 01:00:46.040287 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":981,"thread_start_us":287,"threads_started":1}
I20260806 01:00:46.041522 5610 heartbeater.cc:464] Registering TS with master...
I20260806 01:00:46.041615 5610 heartbeater.cc:511] Master 127.23.242.62:44869 requested a full tablet report, sending...
W20260806 01:00:46.042796 5610 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 01:00:46.534752 5334 heartbeater.cc:464] Registering TS with master...
I20260806 01:00:46.534932 5334 heartbeater.cc:511] Master 127.23.242.62:44869 requested a full tablet report, sending...
I20260806 01:00:46.535545 5133 ts_manager.cc:194] Registered new tserver with Master: fc54ed87a2b44e699d59bf23361339fb (127.23.242.1:38717)
I20260806 01:00:46.536299 5133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.1@KRBTEST.COM'} at 127.23.242.1:48539
Aug 06 01:00:46 dist-test-slave-dhph krb5kdc[4119](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785978046, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:00:46.773962 5472 heartbeater.cc:464] Registering TS with master...
I20260806 01:00:46.774132 5472 heartbeater.cc:511] Master 127.23.242.62:44869 requested a full tablet report, sending...
I20260806 01:00:46.774477 5133 ts_manager.cc:194] Registered new tserver with Master: 8364a3ef9e2843d5a8ef71d5c885ec69 (127.23.242.2:38863)
I20260806 01:00:46.775100 5133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.2@KRBTEST.COM'} at 127.23.242.2:57595
I20260806 01:00:47.043031 5610 heartbeater.cc:464] Registering TS with master...
I20260806 01:00:47.043213 5610 heartbeater.cc:511] Master 127.23.242.62:44869 requested a full tablet report, sending...
I20260806 01:00:47.043505 5133 ts_manager.cc:194] Registered new tserver with Master: 8ff48d3b54894bedb5fab5f076a6230a (127.23.242.3:46191)
I20260806 01:00:47.043989 5133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.3@KRBTEST.COM'} at 127.23.242.3:53867
I20260806 01:00:47.053349 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 01:00:47.064204 5632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:47.054556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41880 (local address 127.23.242.62:44869)
0806 01:00:47.054701 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:47.054704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:47.054781 (+ 77us) server_negotiation.cc:407] Connection header received
0806 01:00:47.054852 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:47.054853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:47.054865 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:47.054964 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:47.055722 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:47.056522 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:47.057597 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:47.057912 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:47.057979 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:47.059572 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:47.059573 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:47.059573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:47.059581 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:47.061801 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:47.062459 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:47.062461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:47.062461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:47.062544 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:47.063027 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:47.063029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:47.063029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:47.063189 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:47.063341 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:47.063781 (+ 440us) server_negotiation.cc:299] Negotiation successful
0806 01:00:47.064031 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260806 01:00:47.067000 5133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
Aug 06 01:00:47 dist-test-slave-dhph krb5kdc[4119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978046, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for HTTP/127.23.242.20@KRBTEST.COM
W20260806 01:00:47.069595 5133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 01:00:47.082046 5640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:47.073221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:38717 (local address 127.0.0.1:35016)
0806 01:00:47.074776 (+ 1555us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:47.074820 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:47.074899 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:47.075200 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:47.075203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:47.075206 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:00:47.075278 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:47.075281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:47.076876 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:47.076877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:47.080041 (+ 3164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:47.080044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:47.081603 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:47.081604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:47.081781 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:47.081805 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 01:00:47.081886 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 01:00:47.081951 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1524,"thread_start_us":93,"threads_started":1}
I20260806 01:00:47.082139 5639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:47.073489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35016 (local address 127.23.242.1:38717)
0806 01:00:47.074705 (+ 1216us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:47.074709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:47.074978 (+ 269us) server_negotiation.cc:407] Connection header received
0806 01:00:47.075014 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:47.075016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:47.075032 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:47.075096 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:00:47.075354 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:47.076787 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:47.080227 (+ 3440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:47.081362 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:47.081542 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:47.081713 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:47.081916 (+ 203us) server_negotiation.cc:299] Negotiation successful
0806 01:00:47.081981 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1185,"thread_start_us":60,"threads_started":1}
I20260806 01:00:47.083108 5252 tablet_service.cc:1467] Processing CreateTablet for tablet 1f43e3ffd54b4574a5c914bb57fd99cc (DEFAULT_TABLE table=test-table [id=623e19bb33084e87aa59d0443e95b630]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:00:47.083545 5252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f43e3ffd54b4574a5c914bb57fd99cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:00:47.088063 5641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:47.073694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33446 (local address 127.23.242.2:38863)
0806 01:00:47.074575 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:47.074605 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:47.082540 (+ 7935us) server_negotiation.cc:407] Connection header received
0806 01:00:47.082614 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:47.082624 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:47.082645 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:47.082745 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:00:47.083063 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:47.084438 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:47.086454 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:47.087481 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:47.087583 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:47.087663 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:47.087916 (+ 253us) server_negotiation.cc:299] Negotiation successful
0806 01:00:47.087964 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":851,"thread_start_us":55,"threads_started":1}
I20260806 01:00:47.088557 5640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:47.073480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:38863 (local address 127.0.0.1:33446)
0806 01:00:47.082455 (+ 8975us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:47.082466 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:47.082515 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:47.082899 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:47.082900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:47.082903 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:00:47.082941 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:47.082942 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:47.084585 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:47.084586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:47.086333 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:47.086337 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:47.087739 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:47.087740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:47.087812 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:47.087815 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 01:00:47.088422 (+ 607us) client_negotiation.cc:240] Negotiation successful
0806 01:00:47.088472 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8962,"thread_start_us":90,"threads_started":1}
I20260806 01:00:47.091552 5403 tablet_service.cc:1467] Processing CreateTablet for tablet 1f43e3ffd54b4574a5c914bb57fd99cc (DEFAULT_TABLE table=test-table [id=623e19bb33084e87aa59d0443e95b630]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:00:47.091960 5403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f43e3ffd54b4574a5c914bb57fd99cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:00:47.095362 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:47.073280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:46191 (local address 127.0.0.1:60300)
0806 01:00:47.078690 (+ 5410us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:47.078703 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:47.078818 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:47.079192 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:47.079193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:47.079196 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:00:47.079235 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:47.079236 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:47.086312 (+ 7076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:47.086314 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:47.093078 (+ 6764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:47.093084 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:47.095089 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:47.095091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:47.095166 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:47.095181 (+ 15us) client_negotiation.cc:769] Sending connection context
0806 01:00:47.095227 (+ 46us) client_negotiation.cc:240] Negotiation successful
0806 01:00:47.095276 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5388,"thread_start_us":89,"threads_started":1}
I20260806 01:00:47.095690 5642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:47.073848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60300 (local address 127.23.242.3:46191)
0806 01:00:47.074442 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:47.074448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:47.078784 (+ 4336us) server_negotiation.cc:407] Connection header received
0806 01:00:47.078912 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:47.078915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:47.078934 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:47.078997 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:00:47.085081 (+ 6084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:47.086162 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:47.093214 (+ 7052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:47.094156 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:47.094243 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:47.094301 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:47.095523 (+ 1222us) server_negotiation.cc:299] Negotiation successful
0806 01:00:47.095574 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":53,"threads_started":1}
I20260806 01:00:47.096323 5539 tablet_service.cc:1467] Processing CreateTablet for tablet 1f43e3ffd54b4574a5c914bb57fd99cc (DEFAULT_TABLE table=test-table [id=623e19bb33084e87aa59d0443e95b630]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:00:47.096676 5539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f43e3ffd54b4574a5c914bb57fd99cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:00:47.100339 5645 tablet_bootstrap.cc:492] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8ff48d3b54894bedb5fab5f076a6230a: Bootstrap starting.
I20260806 01:00:47.101330 5645 tablet_bootstrap.cc:654] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8ff48d3b54894bedb5fab5f076a6230a: Neither blocks nor log segments found. Creating new log.
I20260806 01:00:47.101696 5645 log.cc:824] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8ff48d3b54894bedb5fab5f076a6230a: Log is configured to *not* fsync() on all Append() calls
I20260806 01:00:47.102458 5645 tablet_bootstrap.cc:492] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8ff48d3b54894bedb5fab5f076a6230a: No bootstrap required, opened a new log
I20260806 01:00:47.102528 5645 ts_tablet_manager.cc:1397] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8ff48d3b54894bedb5fab5f076a6230a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 01:00:47.105611 5645 raft_consensus.cc:348] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8ff48d3b54894bedb5fab5f076a6230a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc54ed87a2b44e699d59bf23361339fb" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 38717 } } peers { permanent_uuid: "8ff48d3b54894bedb5fab5f076a6230a" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 46191 } } peers { permanent_uuid: "8364a3ef9e2843d5a8ef71d5c885ec69" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38863 } }
I20260806 01:00:47.105820 5645 raft_consensus.cc:374] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8ff48d3b54894bedb5fab5f076a6230a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:00:47.105859 5645 raft_consensus.cc:729] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8ff48d3b54894bedb5fab5f076a6230a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ff48d3b54894bedb5fab5f076a6230a, State: Initialized, Role: FOLLOWER
I20260806 01:00:47.107349 5645 consensus_queue.cc:260] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8ff48d3b54894bedb5fab5f076a6230a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc54ed87a2b44e699d59bf23361339fb" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 38717 } } peers { permanent_uuid: "8ff48d3b54894bedb5fab5f076a6230a" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 46191 } } peers { permanent_uuid: "8364a3ef9e2843d5a8ef71d5c885ec69" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38863 } }
I20260806 01:00:47.107818 5645 ts_tablet_manager.cc:1428] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8ff48d3b54894bedb5fab5f076a6230a: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260806 01:00:47.107952 5610 heartbeater.cc:503] Master 127.23.242.62:44869 was elected leader, sending a full tablet report...
I20260806 01:00:47.109229 5644 tablet_bootstrap.cc:492] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69: Bootstrap starting.
I20260806 01:00:47.110054 5644 tablet_bootstrap.cc:654] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69: Neither blocks nor log segments found. Creating new log.
I20260806 01:00:47.110555 5644 log.cc:824] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69: Log is configured to *not* fsync() on all Append() calls
I20260806 01:00:47.111788 5643 tablet_bootstrap.cc:492] T 1f43e3ffd54b4574a5c914bb57fd99cc P fc54ed87a2b44e699d59bf23361339fb: Bootstrap starting.
I20260806 01:00:47.113273 5643 tablet_bootstrap.cc:654] T 1f43e3ffd54b4574a5c914bb57fd99cc P fc54ed87a2b44e699d59bf23361339fb: Neither blocks nor log segments found. Creating new log.
I20260806 01:00:47.113686 5643 log.cc:824] T 1f43e3ffd54b4574a5c914bb57fd99cc P fc54ed87a2b44e699d59bf23361339fb: Log is configured to *not* fsync() on all Append() calls
I20260806 01:00:47.114818 5644 tablet_bootstrap.cc:492] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69: No bootstrap required, opened a new log
I20260806 01:00:47.114900 5644 ts_tablet_manager.cc:1397] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260806 01:00:47.118016 5644 raft_consensus.cc:348] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc54ed87a2b44e699d59bf23361339fb" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 38717 } } peers { permanent_uuid: "8ff48d3b54894bedb5fab5f076a6230a" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 46191 } } peers { permanent_uuid: "8364a3ef9e2843d5a8ef71d5c885ec69" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38863 } }
I20260806 01:00:47.118229 5644 raft_consensus.cc:374] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:00:47.118273 5644 raft_consensus.cc:729] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8364a3ef9e2843d5a8ef71d5c885ec69, State: Initialized, Role: FOLLOWER
I20260806 01:00:47.118413 5644 consensus_queue.cc:260] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc54ed87a2b44e699d59bf23361339fb" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 38717 } } peers { permanent_uuid: "8ff48d3b54894bedb5fab5f076a6230a" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 46191 } } peers { permanent_uuid: "8364a3ef9e2843d5a8ef71d5c885ec69" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38863 } }
I20260806 01:00:47.118819 5644 ts_tablet_manager.cc:1428] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260806 01:00:47.119105 5472 heartbeater.cc:503] Master 127.23.242.62:44869 was elected leader, sending a full tablet report...
I20260806 01:00:47.120399 5643 tablet_bootstrap.cc:492] T 1f43e3ffd54b4574a5c914bb57fd99cc P fc54ed87a2b44e699d59bf23361339fb: No bootstrap required, opened a new log
I20260806 01:00:47.120496 5643 ts_tablet_manager.cc:1397] T 1f43e3ffd54b4574a5c914bb57fd99cc P fc54ed87a2b44e699d59bf23361339fb: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260806 01:00:47.123364 5643 raft_consensus.cc:348] T 1f43e3ffd54b4574a5c914bb57fd99cc P fc54ed87a2b44e699d59bf23361339fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc54ed87a2b44e699d59bf23361339fb" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 38717 } } peers { permanent_uuid: "8ff48d3b54894bedb5fab5f076a6230a" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 46191 } } peers { permanent_uuid: "8364a3ef9e2843d5a8ef71d5c885ec69" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38863 } }
I20260806 01:00:47.123553 5643 raft_consensus.cc:374] T 1f43e3ffd54b4574a5c914bb57fd99cc P fc54ed87a2b44e699d59bf23361339fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:00:47.123588 5643 raft_consensus.cc:729] T 1f43e3ffd54b4574a5c914bb57fd99cc P fc54ed87a2b44e699d59bf23361339fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc54ed87a2b44e699d59bf23361339fb, State: Initialized, Role: FOLLOWER
I20260806 01:00:47.123765 5643 consensus_queue.cc:260] T 1f43e3ffd54b4574a5c914bb57fd99cc P fc54ed87a2b44e699d59bf23361339fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc54ed87a2b44e699d59bf23361339fb" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 38717 } } peers { permanent_uuid: "8ff48d3b54894bedb5fab5f076a6230a" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 46191 } } peers { permanent_uuid: "8364a3ef9e2843d5a8ef71d5c885ec69" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38863 } }
I20260806 01:00:47.124191 5643 ts_tablet_manager.cc:1428] T 1f43e3ffd54b4574a5c914bb57fd99cc P fc54ed87a2b44e699d59bf23361339fb: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260806 01:00:47.124382 5334 heartbeater.cc:503] Master 127.23.242.62:44869 was elected leader, sending a full tablet report...
W20260806 01:00:47.258360 5473 tablet.cc:2369] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 01:00:47.265398 5335 tablet.cc:2369] T 1f43e3ffd54b4574a5c914bb57fd99cc P fc54ed87a2b44e699d59bf23361339fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 01:00:47.271842 5611 tablet.cc:2369] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8ff48d3b54894bedb5fab5f076a6230a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 01:00:47.388800 5650 raft_consensus.cc:482] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 01:00:47.389004 5650 raft_consensus.cc:504] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc54ed87a2b44e699d59bf23361339fb" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 38717 } } peers { permanent_uuid: "8ff48d3b54894bedb5fab5f076a6230a" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 46191 } } peers { permanent_uuid: "8364a3ef9e2843d5a8ef71d5c885ec69" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38863 } }
I20260806 01:00:47.389510 5650 leader_election.cc:288] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc54ed87a2b44e699d59bf23361339fb (127.23.242.1:38717), 8ff48d3b54894bedb5fab5f076a6230a (127.23.242.3:46191)
I20260806 01:00:47.394083 5642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:47.390026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:39303 (local address 127.23.242.3:46191)
0806 01:00:47.390071 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:47.390074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:47.390228 (+ 154us) server_negotiation.cc:407] Connection header received
0806 01:00:47.390328 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:47.390329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:47.390340 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:47.390367 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:00:47.390571 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:47.391306 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:47.393030 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:47.393567 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:47.393620 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:47.393630 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:47.393962 (+ 332us) server_negotiation.cc:299] Negotiation successful
0806 01:00:47.394008 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 01:00:47.394380 5653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:47.389753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:46191 (local address 127.23.242.2:39303)
0806 01:00:47.390160 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:47.390171 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:47.390256 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:47.390431 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:47.390432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:47.390434 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:00:47.390508 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:47.390510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:47.391411 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:47.391412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:47.392585 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:47.392588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:47.393782 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:47.393783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:47.393876 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:47.393880 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 01:00:47.394232 (+ 352us) client_negotiation.cc:240] Negotiation successful
0806 01:00:47.394280 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":147,"threads_started":1}
I20260806 01:00:47.394810 5560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f43e3ffd54b4574a5c914bb57fd99cc" candidate_uuid: "8364a3ef9e2843d5a8ef71d5c885ec69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ff48d3b54894bedb5fab5f076a6230a" is_pre_election: true
I20260806 01:00:47.394994 5560 raft_consensus.cc:2454] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8ff48d3b54894bedb5fab5f076a6230a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8364a3ef9e2843d5a8ef71d5c885ec69 in term 0.
I20260806 01:00:47.395256 5356 leader_election.cc:302] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8364a3ef9e2843d5a8ef71d5c885ec69, 8ff48d3b54894bedb5fab5f076a6230a; no voters:
I20260806 01:00:47.395390 5650 raft_consensus.cc:2781] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 01:00:47.395430 5650 raft_consensus.cc:482] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 01:00:47.395452 5650 raft_consensus.cc:3037] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:00:47.396559 5650 raft_consensus.cc:504] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc54ed87a2b44e699d59bf23361339fb" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 38717 } } peers { permanent_uuid: "8ff48d3b54894bedb5fab5f076a6230a" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 46191 } } peers { permanent_uuid: "8364a3ef9e2843d5a8ef71d5c885ec69" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38863 } }
I20260806 01:00:47.396726 5650 leader_election.cc:288] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69 [CANDIDATE]: Term 1 election: Requested vote from peers fc54ed87a2b44e699d59bf23361339fb (127.23.242.1:38717), 8ff48d3b54894bedb5fab5f076a6230a (127.23.242.3:46191)
I20260806 01:00:47.397204 5560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f43e3ffd54b4574a5c914bb57fd99cc" candidate_uuid: "8364a3ef9e2843d5a8ef71d5c885ec69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ff48d3b54894bedb5fab5f076a6230a"
I20260806 01:00:47.397295 5560 raft_consensus.cc:3037] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8ff48d3b54894bedb5fab5f076a6230a [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:00:47.398347 5560 raft_consensus.cc:2454] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8ff48d3b54894bedb5fab5f076a6230a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8364a3ef9e2843d5a8ef71d5c885ec69 in term 1.
I20260806 01:00:47.398610 5356 leader_election.cc:302] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8364a3ef9e2843d5a8ef71d5c885ec69, 8ff48d3b54894bedb5fab5f076a6230a; no voters:
I20260806 01:00:47.398869 5650 raft_consensus.cc:2781] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:00:47.399049 5650 raft_consensus.cc:686] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69 [term 1 LEADER]: Becoming Leader. State: Replica: 8364a3ef9e2843d5a8ef71d5c885ec69, State: Running, Role: LEADER
I20260806 01:00:47.399153 5650 consensus_queue.cc:237] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc54ed87a2b44e699d59bf23361339fb" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 38717 } } peers { permanent_uuid: "8ff48d3b54894bedb5fab5f076a6230a" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 46191 } } peers { permanent_uuid: "8364a3ef9e2843d5a8ef71d5c885ec69" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38863 } }
I20260806 01:00:47.400061 5133 catalog_manager.cc:5495] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8364a3ef9e2843d5a8ef71d5c885ec69 (127.23.242.2). New cstate: current_term: 1 leader_uuid: "8364a3ef9e2843d5a8ef71d5c885ec69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc54ed87a2b44e699d59bf23361339fb" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 38717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ff48d3b54894bedb5fab5f076a6230a" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 46191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8364a3ef9e2843d5a8ef71d5c885ec69" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38863 } health_report { overall_health: HEALTHY } } }
I20260806 01:00:47.400177 5639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:47.390115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:51489 (local address 127.23.242.1:38717)
0806 01:00:47.390164 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:47.390166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:47.390424 (+ 258us) server_negotiation.cc:407] Connection header received
0806 01:00:47.390537 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:47.390538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:47.390548 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:47.390615 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:00:47.390978 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:47.391770 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:47.393510 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:47.394116 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:47.394172 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:47.394190 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:47.400029 (+ 5839us) server_negotiation.cc:299] Negotiation successful
0806 01:00:47.400100 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 01:00:47.401196 5652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:47.389755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:38717 (local address 127.23.242.2:51489)
0806 01:00:47.390349 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:47.390361 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:47.390454 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:47.390692 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:47.390693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:47.390694 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:00:47.390877 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:47.390880 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:47.391886 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:47.391887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:47.393394 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:47.393398 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:47.399810 (+ 6412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:47.399811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:47.399913 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:47.399919 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 01:00:47.400776 (+ 857us) client_negotiation.cc:240] Negotiation successful
0806 01:00:47.400885 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":194,"threads_started":1}
I20260806 01:00:47.402055 5283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f43e3ffd54b4574a5c914bb57fd99cc" candidate_uuid: "8364a3ef9e2843d5a8ef71d5c885ec69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc54ed87a2b44e699d59bf23361339fb" is_pre_election: true
I20260806 01:00:47.402267 5283 raft_consensus.cc:2454] T 1f43e3ffd54b4574a5c914bb57fd99cc P fc54ed87a2b44e699d59bf23361339fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8364a3ef9e2843d5a8ef71d5c885ec69 in term 0.
I20260806 01:00:47.402472 5282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f43e3ffd54b4574a5c914bb57fd99cc" candidate_uuid: "8364a3ef9e2843d5a8ef71d5c885ec69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc54ed87a2b44e699d59bf23361339fb"
I20260806 01:00:47.402555 5282 raft_consensus.cc:3037] T 1f43e3ffd54b4574a5c914bb57fd99cc P fc54ed87a2b44e699d59bf23361339fb [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:00:47.403601 5282 raft_consensus.cc:2454] T 1f43e3ffd54b4574a5c914bb57fd99cc P fc54ed87a2b44e699d59bf23361339fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8364a3ef9e2843d5a8ef71d5c885ec69 in term 1.
I20260806 01:00:47.433511 5641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:47.429537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59404 (local address 127.23.242.2:38863)
0806 01:00:47.429595 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:47.429598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:47.429609 (+ 11us) server_negotiation.cc:407] Connection header received
0806 01:00:47.429660 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:47.429661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:47.429681 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:47.429771 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 01:00:47.429966 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:47.430713 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:47.431864 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:47.432286 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:47.432370 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:47.432595 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 01:00:47.432785 (+ 190us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 01:00:47.433213 (+ 428us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:47.433254 (+ 41us) server_negotiation.cc:299] Negotiation successful
0806 01:00:47.433390 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260806 01:00:47.436481 5282 raft_consensus.cc:1261] T 1f43e3ffd54b4574a5c914bb57fd99cc P fc54ed87a2b44e699d59bf23361339fb [term 1 FOLLOWER]: Refusing update from remote peer 8364a3ef9e2843d5a8ef71d5c885ec69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:00:47.436481 5560 raft_consensus.cc:1261] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8ff48d3b54894bedb5fab5f076a6230a [term 1 FOLLOWER]: Refusing update from remote peer 8364a3ef9e2843d5a8ef71d5c885ec69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:00:47.436933 5650 consensus_queue.cc:1035] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc54ed87a2b44e699d59bf23361339fb" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 38717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:00:47.437284 5650 consensus_queue.cc:1035] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ff48d3b54894bedb5fab5f076a6230a" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 46191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:00:47.446384 5658 mvcc.cc:205] Tried to move back new op lower bound from 7315366082296233984 to 7315366082147594240. Current Snapshot: MvccSnapshot[applied={T|T < 7315366082296233984 or (T in {7315366082296233984})}]
I20260806 01:00:47.447146 5659 mvcc.cc:205] Tried to move back new op lower bound from 7315366082296233984 to 7315366082147594240. Current Snapshot: MvccSnapshot[applied={T|T < 7315366082296233984}]
I20260806 01:00:47.449443 5132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 01:00:47.449568 5132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 01:00:47.450464 5132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06df027bb53c4e11aba1e83695bb996c: Sending DeleteTablet for 3 replicas of tablet 1f43e3ffd54b4574a5c914bb57fd99cc
I20260806 01:00:47.450873 5252 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f43e3ffd54b4574a5c914bb57fd99cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:00:47 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:35016
I20260806 01:00:47.451407 5539 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f43e3ffd54b4574a5c914bb57fd99cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:00:47 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:60300
I20260806 01:00:47.451454 5403 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f43e3ffd54b4574a5c914bb57fd99cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:00:47 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:33446
I20260806 01:00:47.452307 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 5172
I20260806 01:00:47.458420 5668 tablet_replica.cc:330] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8ff48d3b54894bedb5fab5f076a6230a: stopping tablet replica
I20260806 01:00:47.458781 5669 tablet_replica.cc:330] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69: stopping tablet replica
I20260806 01:00:47.458910 5669 raft_consensus.cc:2229] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69 [term 1 LEADER]: Raft consensus shutting down.
I20260806 01:00:47.459018 5669 raft_consensus.cc:2258] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 01:00:47.459151 5668 raft_consensus.cc:2229] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8ff48d3b54894bedb5fab5f076a6230a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 01:00:47.459276 5668 raft_consensus.cc:2258] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8ff48d3b54894bedb5fab5f076a6230a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 01:00:47.459829 5669 ts_tablet_manager.cc:1905] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 01:00:47.462179 5669 ts_tablet_manager.cc:1918] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 01:00:47.462337 5669 log.cc:1196] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-1/wal/wals/1f43e3ffd54b4574a5c914bb57fd99cc
I20260806 01:00:47.462818 5669 ts_tablet_manager.cc:1939] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8364a3ef9e2843d5a8ef71d5c885ec69: Deleting consensus metadata
I20260806 01:00:47.463440 5117 catalog_manager.cc:4841] TS 8364a3ef9e2843d5a8ef71d5c885ec69 (127.23.242.2:38863): tablet 1f43e3ffd54b4574a5c914bb57fd99cc (table test-table [id=623e19bb33084e87aa59d0443e95b630]) successfully deleted
I20260806 01:00:47.464708 5668 ts_tablet_manager.cc:1905] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8ff48d3b54894bedb5fab5f076a6230a: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 01:00:47 dist-test-slave-dhph krb5kdc[4119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978046, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for HTTP/127.23.242.20@KRBTEST.COM
I20260806 01:00:47.476840 5668 ts_tablet_manager.cc:1918] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8ff48d3b54894bedb5fab5f076a6230a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 01:00:47.476994 5668 log.cc:1196] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8ff48d3b54894bedb5fab5f076a6230a: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785977869987768-24520-0/minicluster-data/ts-2/wal/wals/1f43e3ffd54b4574a5c914bb57fd99cc
I20260806 01:00:47.477334 5668 ts_tablet_manager.cc:1939] T 1f43e3ffd54b4574a5c914bb57fd99cc P 8ff48d3b54894bedb5fab5f076a6230a: Deleting consensus metadata
W20260806 01:00:47.477862 5118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.242.1:38717 (error 108)
I20260806 01:00:47.479064 5117 catalog_manager.cc:4841] TS 8ff48d3b54894bedb5fab5f076a6230a (127.23.242.3:46191): tablet 1f43e3ffd54b4574a5c914bb57fd99cc (table test-table [id=623e19bb33084e87aa59d0443e95b630]) successfully deleted
I20260806 01:00:47.479611 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:47.479476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:38717 (local address 127.0.0.1:44024)
0806 01:00:47.479531 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:47.479572 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.1:38717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260806 01:00:47.479641 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 5338
W20260806 01:00:47.479844 5118 catalog_manager.cc:4584] TS fc54ed87a2b44e699d59bf23361339fb (127.23.242.1:38717): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1f43e3ffd54b4574a5c914bb57fd99cc: Network error: Client connection negotiation failed: client connection to 127.23.242.1:38717: connect: Connection refused (error 111)
I20260806 01:00:47.488198 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 5475
I20260806 01:00:47.495705 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 5100
I20260806 01:00:47.503090 24520 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 01:00:47.849788 24520 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 01:00:47.849937 24520 mini_ranger.cc:66] Stopping Ranger...
I20260806 01:00:48.196378 24520 mini_ranger.cc:68] Stopped Ranger
2026-08-06 01:00:48.196 UTC [4150] LOG: received smart shutdown request
2026-08-06 01:00:48.201 UTC [4150] LOG: background worker "logical replication launcher" (PID 4159) exited with exit code 1
2026-08-06 01:00:48.202 UTC [4154] LOG: shutting down
2026-08-06 01:00:48.439 UTC [4150] LOG: database system is shut down
2026-08-06T01:00:48Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (58186 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 01:00:48 dist-test-slave-dhph krb5kdc[5681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 01:00:48 dist-test-slave-dhph krb5kdc[5681](info): set up 2 sockets
Aug 06 01:00:48 dist-test-slave-dhph krb5kdc[5681](info): commencing operation
krb5kdc: starting...
W20260806 01:00:49.826131 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.218s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 01:00:49 dist-test-slave-dhph krb5kdc[5681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T01:00:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:00:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 01:00:49.987550 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:44525
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:44525,127.23.242.61:45479,127.23.242.60:34743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:00:50.083563 5694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:00:50.083813 5694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:00:50.083855 5694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:00:50.085019 5694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:00:50.085090 5694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:00:50.085115 5694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:00:50.085137 5694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:00:50.085166 5694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:00:50.087177 5694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:44525,127.23.242.61:45479,127.23.242.60:34743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:44525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:00:50.087667 5694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:00:50.087960 5694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:00:50.093690 5703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:50.093878 5700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:50.093904 5699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:50.094497 5694 server_base.cc:1034] running on GCE node
I20260806 01:00:50.094835 5694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:00:50.095115 5694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:00:50.096330 5694 hybrid_clock.cc:648] HybridClock initialized: now 1785978050096318 us; error 61 us; skew 500 ppm
Aug 06 01:00:50 dist-test-slave-dhph krb5kdc[5681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978050, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:00:50.103222 5694 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 01:00:50.103828 5694 webserver.cc:460] Webserver started at http://127.23.242.62:41445/ using document root <none> and password file <none>
I20260806 01:00:50.104076 5694 fs_manager.cc:359] Metadata directory not provided
I20260806 01:00:50.104125 5694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:00:50.104256 5694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:00:50.105406 5694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "c9ee7af2dc21485e89cf198997caa562"
format_stamp: "Formatted at 2026-08-06 01:00:50 on dist-test-slave-dhph"
server_key: "742e7ac4a34fb2ac29d3b114aed5e777"
server_key_iv: "dd5340fa8ad7d89f0ccc8f7c26db3b6f"
server_key_version: "encryptionkey@0"
I20260806 01:00:50.105791 5694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "c9ee7af2dc21485e89cf198997caa562"
format_stamp: "Formatted at 2026-08-06 01:00:50 on dist-test-slave-dhph"
server_key: "742e7ac4a34fb2ac29d3b114aed5e777"
server_key_iv: "dd5340fa8ad7d89f0ccc8f7c26db3b6f"
server_key_version: "encryptionkey@0"
I20260806 01:00:50.107421 5694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 01:00:50.108667 5710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:50.109036 5694 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:00:50.109180 5694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "c9ee7af2dc21485e89cf198997caa562"
format_stamp: "Formatted at 2026-08-06 01:00:50 on dist-test-slave-dhph"
server_key: "742e7ac4a34fb2ac29d3b114aed5e777"
server_key_iv: "dd5340fa8ad7d89f0ccc8f7c26db3b6f"
server_key_version: "encryptionkey@0"
I20260806 01:00:50.109282 5694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:00:50.148161 5694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:00:50.151013 5694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:00:50.151239 5694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:00:50.157248 5694 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:44525
I20260806 01:00:50.157645 5763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:44525 every 8 connection(s)
I20260806 01:00:50.157697 5694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:00:50.158783 5764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:00:50.160607 5764 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44525 } has no permanent_uuid. Determining permanent_uuid...
I20260806 01:00:50.166209 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 5694
I20260806 01:00:50.166663 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:00:50.166955 24520 external_mini_cluster.cc:1428] Setting key 5e0450ee8965988603f99b3e84ffcd5d
Aug 06 01:00:50 dist-test-slave-dhph krb5kdc[5681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978050, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:00:50.172438 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.161465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:44525 (local address 127.0.0.1:39406)
0806 01:00:50.161677 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:50.161771 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:50.162304 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:50.162635 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:50.162638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:50.162962 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:00:50.163123 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:50.163128 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.164165 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.164167 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:50.164875 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:50.164877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.165404 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.165405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:50.165476 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:50.166175 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:00:50.166184 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:00:50.168005 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:00:50.170211 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:50.170231 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:50.170235 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:50.170788 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:50.171262 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:50.171263 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:50.171264 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:50.171377 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:50.171929 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:00:50.171938 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:00:50.172014 (+ 76us) client_negotiation.cc:769] Sending connection context
0806 01:00:50.172111 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 01:00:50.172262 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":83,"threads_started":1}
I20260806 01:00:50.172469 5767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.161646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39406 (local address 127.23.242.62:44525)
0806 01:00:50.161786 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:50.161792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:50.161883 (+ 91us) server_negotiation.cc:407] Connection header received
0806 01:00:50.162466 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:50.162468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:50.162488 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:50.162616 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:50.163235 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.164049 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.164968 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.165302 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.165430 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:50.168104 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:50.168114 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:50.168114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:50.168123 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:50.170043 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.170956 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.170961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.170961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.171138 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.171475 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.171476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.171476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.171811 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:50.171914 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:50.172202 (+ 288us) server_negotiation.cc:299] Negotiation successful
0806 01:00:50.172357 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":113,"thread_start_us":46,"threads_started":1}
I20260806 01:00:50.173593 5764 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45479 } has no permanent_uuid. Determining permanent_uuid...
I20260806 01:00:50.173982 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.173857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:45479 (local address 127.0.0.1:45554)
0806 01:00:50.173910 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:50.173960 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.61:45479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260806 01:00:50.174124 5711 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.242.61:45479: connect: Connection refused (error 111)
I20260806 01:00:50.174621 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.174488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:45479 (local address 127.0.0.1:45568)
0806 01:00:50.174547 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:50.174572 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.61:45479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 01:00:50.174744 5764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.242.61:45479: Network error: Client connection negotiation failed: client connection to 127.23.242.61:45479: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.23.242.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.61.keytab.
Entry for principal kudu/127.23.242.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.61.keytab.
I20260806 01:00:50.200121 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.61:45479
--webserver_interface=127.23.242.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.61.keytab
--principal=kudu/127.23.242.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:44525,127.23.242.61:45479,127.23.242.60:34743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 01:00:50.223984 5764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45479 } attempt: 1
I20260806 01:00:50.224617 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.224419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:45479 (local address 127.0.0.1:45578)
0806 01:00:50.224491 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:50.224545 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.61:45479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 01:00:50.225099 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.224995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:45479 (local address 127.0.0.1:45586)
0806 01:00:50.225043 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:50.225076 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.61:45479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260806 01:00:50.225221 5764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.242.61:45479: Network error: Client connection negotiation failed: client connection to 127.23.242.61:45479: connect: Connection refused (error 111)
I20260806 01:00:50.293442 5764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45479 } attempt: 2
I20260806 01:00:50.294035 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.293879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:45479 (local address 127.0.0.1:45598)
0806 01:00:50.293951 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:50.293997 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.61:45479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260806 01:00:50.294607 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.294450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:45479 (local address 127.0.0.1:45606)
0806 01:00:50.294508 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:50.294546 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.61:45479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260806 01:00:50.294711 5764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.242.61:45479: Network error: Client connection negotiation failed: client connection to 127.23.242.61:45479: connect: Connection refused (error 111)
W20260806 01:00:50.295601 5771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:00:50.295809 5771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:00:50.295835 5771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:00:50.296972 5771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:00:50.297013 5771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:00:50.297029 5771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:00:50.297042 5771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:00:50.297055 5771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:00:50.298645 5771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:44525,127.23.242.61:45479,127.23.242.60:34743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.61.keytab
--principal=kudu/127.23.242.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.61:45479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.242.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:00:50.298877 5771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:00:50.299063 5771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:00:50.304736 5779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:50.304849 5771 server_base.cc:1034] running on GCE node
W20260806 01:00:50.304744 5776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:50.304757 5777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:50.305380 5771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:00:50.305740 5771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:00:50.306952 5771 hybrid_clock.cc:648] HybridClock initialized: now 1785978050306866 us; error 104 us; skew 500 ppm
Aug 06 01:00:50 dist-test-slave-dhph krb5kdc[5681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978050, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:00:50.310161 5771 init.cc:377] Logged in from keytab as kudu/127.23.242.61@KRBTEST.COM (short username kudu)
I20260806 01:00:50.310776 5771 webserver.cc:460] Webserver started at http://127.23.242.61:44335/ using document root <none> and password file <none>
I20260806 01:00:50.310978 5771 fs_manager.cc:359] Metadata directory not provided
I20260806 01:00:50.311021 5771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:00:50.311113 5771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:00:50.312593 5771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-1/data/instance:
uuid: "c0d6dd9a35e940e092e44e0887409a1d"
format_stamp: "Formatted at 2026-08-06 01:00:50 on dist-test-slave-dhph"
server_key: "dcdd29118c1e505fba27e090d811f4c5"
server_key_iv: "0d8b567ec96fc07ed51f99bdb7c6fda2"
server_key_version: "encryptionkey@0"
I20260806 01:00:50.313129 5771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-1/wal/instance:
uuid: "c0d6dd9a35e940e092e44e0887409a1d"
format_stamp: "Formatted at 2026-08-06 01:00:50 on dist-test-slave-dhph"
server_key: "dcdd29118c1e505fba27e090d811f4c5"
server_key_iv: "0d8b567ec96fc07ed51f99bdb7c6fda2"
server_key_version: "encryptionkey@0"
I20260806 01:00:50.315094 5771 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260806 01:00:50.316203 5786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:50.316401 5771 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:00:50.316490 5771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-1/wal
uuid: "c0d6dd9a35e940e092e44e0887409a1d"
format_stamp: "Formatted at 2026-08-06 01:00:50 on dist-test-slave-dhph"
server_key: "dcdd29118c1e505fba27e090d811f4c5"
server_key_iv: "0d8b567ec96fc07ed51f99bdb7c6fda2"
server_key_version: "encryptionkey@0"
I20260806 01:00:50.316591 5771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:00:50.328092 5771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:00:50.330940 5771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:00:50.331077 5771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:00:50.335618 5771 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.61:45479
I20260806 01:00:50.336120 5771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-1/data/info.pb
I20260806 01:00:50.336261 5840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.61:45479 every 8 connection(s)
I20260806 01:00:50.337090 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 5771
I20260806 01:00:50.337203 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-1/wal/instance
I20260806 01:00:50.337384 5841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:00:50.337419 24520 external_mini_cluster.cc:1428] Setting key f6f7033ba6347a75900dcabaf23bdeef
I20260806 01:00:50.341017 5841 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44525 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.242.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.60@KRBTEST.COM" created.
Aug 06 01:00:50 dist-test-slave-dhph krb5kdc[5681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978050, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.61@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:00:50.356455 5767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.342032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39408 (local address 127.23.242.62:44525)
0806 01:00:50.342114 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:50.342116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:50.342401 (+ 285us) server_negotiation.cc:407] Connection header received
0806 01:00:50.343046 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:50.343048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:50.343073 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:50.343153 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:50.343932 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.344773 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.346832 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.347216 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.347300 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:50.351629 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:50.351631 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:50.351631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:50.351646 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:50.353884 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.355055 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.355056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.355057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.355166 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.355613 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.355614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.355614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.355783 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:50.355864 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:50.356197 (+ 333us) server_negotiation.cc:299] Negotiation successful
0806 01:00:50.356309 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 01:00:50.356666 5844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.341996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:44525 (local address 127.0.0.1:39408)
0806 01:00:50.342252 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:50.342300 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:50.342932 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:50.343269 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:50.343273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:50.343580 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:00:50.343825 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:50.343830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.344904 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.344905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:50.346733 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:50.346736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.347337 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.347339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:50.347638 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:50.348192 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:00:50.348201 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:00:50.351513 (+ 3312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:00:50.354076 (+ 2563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:50.354095 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:50.354098 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:50.354698 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:50.355320 (+ 622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:50.355322 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:50.355322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:50.355517 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:50.355868 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:00:50.355875 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:00:50.356026 (+ 151us) client_negotiation.cc:769] Sending connection context
0806 01:00:50.356127 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 01:00:50.356404 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":164,"threads_started":1}
I20260806 01:00:50.357635 5841 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45479 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 01:00:50 dist-test-slave-dhph krb5kdc[5681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978050, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.61@KRBTEST.COM for kudu/127.23.242.61@KRBTEST.COM
I20260806 01:00:50.368808 5844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.358101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:45479 (local address 127.0.0.1:45612)
0806 01:00:50.358159 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:50.358170 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:50.358214 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:50.358483 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:50.358484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:50.358701 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:00:50.358753 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:50.358754 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.359620 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.359621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:50.360118 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:50.360120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.360574 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.360574 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:50.360649 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:50.361100 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:00:50.361105 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:00:50.365126 (+ 4021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:00:50.367321 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:50.367322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:50.367323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:50.367493 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:50.367784 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:50.367784 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:50.367785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:50.367850 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:50.368235 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:00:50.368236 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:00:50.368280 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 01:00:50.368367 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 01:00:50.368632 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260806 01:00:50.369060 5846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.358176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45612 (local address 127.23.242.61:45479)
0806 01:00:50.358349 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:50.358352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:50.358370 (+ 18us) server_negotiation.cc:407] Connection header received
0806 01:00:50.358387 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:50.358389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:50.358399 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:50.358483 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:50.358831 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.359520 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.360199 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.360499 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.360578 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:50.365246 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:50.365259 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:50.365259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:50.365267 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:50.367217 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.367616 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.367620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.367621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.367696 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.367921 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.367922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.367922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.368092 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:50.368257 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:50.368401 (+ 144us) server_negotiation.cc:299] Negotiation successful
0806 01:00:50.368804 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":221,"server-negotiator.queue_time_us":145,"thread_start_us":91,"threads_started":1}
I20260806 01:00:50.370090 5841 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } has no permanent_uuid. Determining permanent_uuid...
I20260806 01:00:50.370568 5844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.370386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:34743 (local address 127.0.0.1:37074)
0806 01:00:50.370484 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:50.370535 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260806 01:00:50.370820 5788 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.242.60:34743: connect: Connection refused (error 111)
I20260806 01:00:50.371461 5844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.371332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:34743 (local address 127.0.0.1:37090)
0806 01:00:50.371402 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:50.371436 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 01:00:50.371637 5841 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.242.60:34743: Network error: Client connection negotiation failed: client connection to 127.23.242.60:34743: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.60.keytab.
Entry for principal kudu/127.23.242.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.60.keytab.
I20260806 01:00:50.375146 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.60:34743
--webserver_interface=127.23.242.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.60.keytab
--principal=kudu/127.23.242.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:44525,127.23.242.61:45479,127.23.242.60:34743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 01:00:50.385869 5764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45479 } attempt: 3
Aug 06 01:00:50 dist-test-slave-dhph krb5kdc[5681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978050, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for kudu/127.23.242.61@KRBTEST.COM
I20260806 01:00:50.393543 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.386338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:45479 (local address 127.0.0.1:45624)
0806 01:00:50.386417 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:50.386434 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:50.386510 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:50.386746 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:50.386747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:50.386910 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:00:50.386974 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:50.386976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.387688 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.387689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:50.388294 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:50.388297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.388767 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.388768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:50.388858 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:50.389387 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:00:50.389393 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:00:50.390765 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:00:50.392483 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:50.392484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:50.392485 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:50.392659 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:50.392876 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:50.392876 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:50.392877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:50.392941 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:50.393196 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:00:50.393197 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:00:50.393248 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 01:00:50.393337 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 01:00:50.393440 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 01:00:50.393543 5846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.386449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45624 (local address 127.23.242.61:45479)
0806 01:00:50.386516 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:50.386518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:50.386528 (+ 10us) server_negotiation.cc:407] Connection header received
0806 01:00:50.386596 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:50.386597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:50.386618 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:50.386736 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:50.387070 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.387600 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.388413 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.388686 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.388770 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:50.390860 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:50.390861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:50.390862 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:50.390872 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:50.392381 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.392749 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.392750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.392751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.392807 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.393006 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.393007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.393008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.393126 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:50.393203 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:50.393362 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 01:00:50.393441 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 01:00:50.394142 5764 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } has no permanent_uuid. Determining permanent_uuid...
I20260806 01:00:50.394542 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.394405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:34743 (local address 127.0.0.1:37092)
0806 01:00:50.394490 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:50.394521 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 01:00:50.394982 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.394898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:34743 (local address 127.0.0.1:37096)
0806 01:00:50.394941 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:50.394965 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260806 01:00:50.395073 5764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.242.60:34743: Network error: Client connection negotiation failed: client connection to 127.23.242.60:34743: connect: Connection refused (error 111)
I20260806 01:00:50.420953 5841 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } attempt: 1
I20260806 01:00:50.421589 5844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.421427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:34743 (local address 127.0.0.1:37112)
0806 01:00:50.421500 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:50.421549 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260806 01:00:50.422065 5844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.421960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:34743 (local address 127.0.0.1:37124)
0806 01:00:50.422008 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:50.422038 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260806 01:00:50.422273 5841 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.242.60:34743: Network error: Client connection negotiation failed: client connection to 127.23.242.60:34743: connect: Connection refused (error 111)
I20260806 01:00:50.426257 5764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } attempt: 1
I20260806 01:00:50.426913 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.426781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:34743 (local address 127.0.0.1:37140)
0806 01:00:50.426846 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:50.426886 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 01:00:50.427240 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.427160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:34743 (local address 127.0.0.1:37148)
0806 01:00:50.427189 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:50.427214 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260806 01:00:50.427336 5764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.242.60:34743: Network error: Client connection negotiation failed: client connection to 127.23.242.60:34743: connect: Connection refused (error 111)
W20260806 01:00:50.469583 5848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:00:50.469864 5848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:00:50.469908 5848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:00:50.471236 5848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:00:50.471321 5848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:00:50.471351 5848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:00:50.471386 5848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:00:50.471417 5848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:00:50.473254 5848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:44525,127.23.242.61:45479,127.23.242.60:34743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.60.keytab
--principal=kudu/127.23.242.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.60:34743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.242.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:00:50.473594 5848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:00:50.473870 5848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:00:50.479918 5853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:50.479923 5854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:50.479923 5856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:50.480346 5848 server_base.cc:1034] running on GCE node
I20260806 01:00:50.480788 5848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:00:50.481189 5848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:00:50.482360 5848 hybrid_clock.cc:648] HybridClock initialized: now 1785978050482334 us; error 41 us; skew 500 ppm
Aug 06 01:00:50 dist-test-slave-dhph krb5kdc[5681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978050, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:00:50.485291 5848 init.cc:377] Logged in from keytab as kudu/127.23.242.60@KRBTEST.COM (short username kudu)
I20260806 01:00:50.485846 5848 webserver.cc:460] Webserver started at http://127.23.242.60:37245/ using document root <none> and password file <none>
I20260806 01:00:50.486119 5848 fs_manager.cc:359] Metadata directory not provided
I20260806 01:00:50.486191 5848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:00:50.486353 5848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:00:50.487509 5848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-2/data/instance:
uuid: "ddcc94a2f600434186509efae6d4dcf9"
format_stamp: "Formatted at 2026-08-06 01:00:50 on dist-test-slave-dhph"
server_key: "32d47d2d7fff532e2620ef03f829a577"
server_key_iv: "d5ff1486226262a12bfbab6d2fbe4e3e"
server_key_version: "encryptionkey@0"
I20260806 01:00:50.487895 5848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-2/wal/instance:
uuid: "ddcc94a2f600434186509efae6d4dcf9"
format_stamp: "Formatted at 2026-08-06 01:00:50 on dist-test-slave-dhph"
server_key: "32d47d2d7fff532e2620ef03f829a577"
server_key_iv: "d5ff1486226262a12bfbab6d2fbe4e3e"
server_key_version: "encryptionkey@0"
I20260806 01:00:50.489315 5848 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260806 01:00:50.490338 5863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:50.490619 5841 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } attempt: 2
I20260806 01:00:50.490720 5848 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:00:50.490804 5848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-2/wal
uuid: "ddcc94a2f600434186509efae6d4dcf9"
format_stamp: "Formatted at 2026-08-06 01:00:50 on dist-test-slave-dhph"
server_key: "32d47d2d7fff532e2620ef03f829a577"
server_key_iv: "d5ff1486226262a12bfbab6d2fbe4e3e"
server_key_version: "encryptionkey@0"
I20260806 01:00:50.490873 5848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:00:50.491145 5844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.490986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:34743 (local address 127.0.0.1:37158)
0806 01:00:50.491061 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:50.491111 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 01:00:50.491554 5844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.491436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:34743 (local address 127.0.0.1:37162)
0806 01:00:50.491490 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:50.491523 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 01:00:50.491722 5841 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.242.60:34743: Network error: Client connection negotiation failed: client connection to 127.23.242.60:34743: connect: Connection refused (error 111)
I20260806 01:00:50.502533 5764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } attempt: 2
I20260806 01:00:50.503124 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.502921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:34743 (local address 127.0.0.1:37176)
0806 01:00:50.503007 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:50.503051 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 01:00:50.503547 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.503435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:34743 (local address 127.0.0.1:37188)
0806 01:00:50.503489 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:50.503526 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:34743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260806 01:00:50.503659 5764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.242.60:34743: Network error: Client connection negotiation failed: client connection to 127.23.242.60:34743: connect: Connection refused (error 111)
I20260806 01:00:50.504288 5848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:00:50.506899 5848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:00:50.507081 5848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:00:50.513615 5848 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.60:34743
I20260806 01:00:50.513724 5919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.60:34743 every 8 connection(s)
I20260806 01:00:50.514174 5848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-2/data/info.pb
I20260806 01:00:50.515296 5920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:00:50.516867 5920 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44525 } has no permanent_uuid. Determining permanent_uuid...
I20260806 01:00:50.522917 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 5848
I20260806 01:00:50.523056 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785977869987768-24520-0/minicluster-data/master-2/wal/instance
I20260806 01:00:50.523260 24520 external_mini_cluster.cc:1428] Setting key 18fe570755d579040c0ac529d2038f5d
Aug 06 01:00:50 dist-test-slave-dhph krb5kdc[5681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978050, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.60@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:00:50.528452 5922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.517547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:44525 (local address 127.0.0.1:39414)
0806 01:00:50.517826 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:50.517865 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:50.518403 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:50.518918 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:50.518922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:50.519257 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:00:50.519643 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:50.519650 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.520363 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.520364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:50.521267 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:50.521271 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.521658 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.521659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:50.521788 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:50.522341 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:00:50.522349 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:00:50.524193 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:00:50.526385 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:50.526391 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:50.526393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:50.526756 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:50.527116 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:50.527117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:50.527118 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:50.527305 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:50.527794 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:00:50.527800 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:00:50.527931 (+ 131us) client_negotiation.cc:769] Sending connection context
0806 01:00:50.528065 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 01:00:50.528256 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":93,"threads_started":1}
I20260806 01:00:50.528488 5767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.517629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39414 (local address 127.23.242.62:44525)
0806 01:00:50.517729 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:50.517731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:50.517953 (+ 222us) server_negotiation.cc:407] Connection header received
0806 01:00:50.518719 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:50.518721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:50.518745 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:50.518830 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:50.519770 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.520260 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.521366 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.521570 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.521649 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:50.524317 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:50.524318 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:50.524319 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:50.524327 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:50.526194 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.526846 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.526848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.526848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.527010 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.527455 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.527456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.527456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.527676 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:50.527763 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:50.528230 (+ 467us) server_negotiation.cc:299] Negotiation successful
0806 01:00:50.528369 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 01:00:50.529227 5920 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45479 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 01:00:50 dist-test-slave-dhph krb5kdc[5681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.60@KRBTEST.COM
Aug 06 01:00:50 dist-test-slave-dhph krb5kdc[5681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978050, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.60@KRBTEST.COM for kudu/127.23.242.61@KRBTEST.COM
I20260806 01:00:50.534148 5923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.523790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37196 (local address 127.23.242.60:34743)
0806 01:00:50.523983 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:50.523992 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:50.524104 (+ 112us) server_negotiation.cc:407] Connection header received
0806 01:00:50.524134 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:50.524136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:50.524156 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:50.524242 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:50.524860 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.525822 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.527194 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.527598 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.527657 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:50.530900 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:50.530908 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:50.530909 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:50.530916 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:50.532723 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.533091 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.533094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.533094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.533157 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.533383 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.533384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.533384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.533538 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:50.533680 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:50.533888 (+ 208us) server_negotiation.cc:299] Negotiation successful
0806 01:00:50.533994 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":90,"threads_started":1}
I20260806 01:00:50.537452 5922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.529559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:45479 (local address 127.0.0.1:45640)
0806 01:00:50.529710 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:50.529723 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:50.529783 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:50.529946 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:50.529947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:50.530066 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:00:50.530115 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:50.530116 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.531019 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.531020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:50.531764 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:50.531766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.532266 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.532267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:50.532330 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:50.532794 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:00:50.532798 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:00:50.534131 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:00:50.536335 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:50.536337 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:50.536337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:50.536528 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:50.536822 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:50.536823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:50.536824 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:50.536885 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:50.537209 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:00:50.537210 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:00:50.537249 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 01:00:50.537308 (+ 59us) client_negotiation.cc:240] Negotiation successful
0806 01:00:50.537387 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260806 01:00:50.537802 5846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.529647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45640 (local address 127.23.242.61:45479)
0806 01:00:50.529716 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:50.529718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:50.529789 (+ 71us) server_negotiation.cc:407] Connection header received
0806 01:00:50.529834 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:50.529835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:50.529847 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:50.529916 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:50.530196 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.530921 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.531867 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.532169 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.532258 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:50.534222 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:50.534224 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:50.534225 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:50.534232 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:50.536243 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.536638 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.536639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.536639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.536702 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.536954 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.536955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.536956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.537132 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:50.537209 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:50.537376 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 01:00:50.537463 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 01:00:50.538213 5920 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 01:00:50 dist-test-slave-dhph krb5kdc[5681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
Aug 06 01:00:50 dist-test-slave-dhph krb5kdc[5681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978050, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.60@KRBTEST.COM for kudu/127.23.242.60@KRBTEST.COM
I20260806 01:00:50.544621 5767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.535183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39424 (local address 127.23.242.62:44525)
0806 01:00:50.535242 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:50.535244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:50.535292 (+ 48us) server_negotiation.cc:407] Connection header received
0806 01:00:50.535375 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:50.535376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:50.535388 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:50.535445 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:50.536196 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.536674 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.537933 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.538215 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.538276 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:50.541356 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:50.541357 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:50.541358 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:50.541365 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:50.543096 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.543537 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.543537 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.543538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.543603 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.543891 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.543892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.543893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.544069 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:50.544230 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:50.544409 (+ 179us) server_negotiation.cc:299] Negotiation successful
0806 01:00:50.544503 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260806 01:00:50.547194 5922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.538507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:34743 (local address 127.0.0.1:37200)
0806 01:00:50.539007 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:50.539021 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:50.539092 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:50.539357 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:50.539359 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:50.539604 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:00:50.539667 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:50.539669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.540768 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.540770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:50.541399 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:50.541402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.541906 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.541907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:50.541967 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:50.542354 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:00:50.542359 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:00:50.543928 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:00:50.545941 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:50.545942 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:50.545943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:50.546190 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:50.546451 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:50.546452 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:50.546453 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:50.546521 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:50.546932 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:00:50.546932 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:00:50.546972 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 01:00:50.547053 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 01:00:50.547128 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469}
I20260806 01:00:50.547343 5923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.538769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37200 (local address 127.23.242.60:34743)
0806 01:00:50.538814 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:50.538816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:50.539142 (+ 326us) server_negotiation.cc:407] Connection header received
0806 01:00:50.539171 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:50.539172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:50.539189 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:50.539270 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:50.539797 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.540654 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.541507 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.541728 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.541907 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:50.544034 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:50.544035 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:50.544036 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:50.544042 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:50.545849 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.546300 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.546302 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.546302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.546369 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.546664 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.546665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.546665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.546816 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:50.546905 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:50.547107 (+ 202us) server_negotiation.cc:299] Negotiation successful
0806 01:00:50.547246 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 01:00:50.548789 5920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9: Bootstrap starting.
I20260806 01:00:50.549708 5920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9: Neither blocks nor log segments found. Creating new log.
I20260806 01:00:50.550011 5920 log.cc:824] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9: Log is configured to *not* fsync() on all Append() calls
I20260806 01:00:50.550951 5920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9: No bootstrap required, opened a new log
Aug 06 01:00:50 dist-test-slave-dhph krb5kdc[5681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.61@KRBTEST.COM
I20260806 01:00:50.552472 5920 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ee7af2dc21485e89cf198997caa562" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44525 } } peers { permanent_uuid: "c0d6dd9a35e940e092e44e0887409a1d" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45479 } } peers { permanent_uuid: "ddcc94a2f600434186509efae6d4dcf9" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } }
I20260806 01:00:50.552634 5920 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:00:50.552692 5920 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddcc94a2f600434186509efae6d4dcf9, State: Initialized, Role: FOLLOWER
I20260806 01:00:50.552834 5920 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ee7af2dc21485e89cf198997caa562" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44525 } } peers { permanent_uuid: "c0d6dd9a35e940e092e44e0887409a1d" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45479 } } peers { permanent_uuid: "ddcc94a2f600434186509efae6d4dcf9" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } }
I20260806 01:00:50.553139 5926 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ee7af2dc21485e89cf198997caa562" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44525 } } peers { permanent_uuid: "c0d6dd9a35e940e092e44e0887409a1d" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45479 } } peers { permanent_uuid: "ddcc94a2f600434186509efae6d4dcf9" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } } }
I20260806 01:00:50.553201 5920 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:00:50.553270 5926 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 01:00:50.555060 5846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.545850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45650 (local address 127.23.242.61:45479)
0806 01:00:50.545920 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:50.545922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:50.545931 (+ 9us) server_negotiation.cc:407] Connection header received
0806 01:00:50.545956 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:50.545957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:50.545967 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:50.546024 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:50.546698 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.547497 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.548338 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.548560 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.548660 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:50.551438 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:50.551440 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:50.551440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:50.551449 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:50.553139 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.553542 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.553543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.553543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.553604 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.553825 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.553826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.553827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.554181 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:50.554701 (+ 520us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:50.554824 (+ 123us) server_negotiation.cc:299] Negotiation successful
0806 01:00:50.554950 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260806 01:00:50.555428 5937 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 01:00:50.555522 5937 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 01:00:50.556777 24520 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 01:00:50.555619 5848 master_runner.cc:186] Error getting master registration for 127.23.242.62:44525: OK, instance_id { permanent_uuid: "c9ee7af2dc21485e89cf198997caa562" instance_seqno: 1785978050154567 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 01:00:50.559239 5848 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 01:00:50.567951 5846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.558919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45654 (local address 127.23.242.61:45479)
0806 01:00:50.559011 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:50.559013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:50.559022 (+ 9us) server_negotiation.cc:407] Connection header received
0806 01:00:50.559046 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:50.559047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:50.559058 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:50.559133 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:50.560044 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.560568 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.561384 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.561612 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.561705 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:50.563531 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:50.563533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:50.563533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:50.563541 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:50.565642 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.566058 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.566060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.566061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.566152 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.566502 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.566503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.566503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.566692 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:50.567348 (+ 656us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:50.567727 (+ 379us) server_negotiation.cc:299] Negotiation successful
0806 01:00:50.567819 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 01:00:50.569492 5767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.558481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39436 (local address 127.23.242.62:44525)
0806 01:00:50.558526 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:50.558528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:50.558968 (+ 440us) server_negotiation.cc:407] Connection header received
0806 01:00:50.559123 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:50.559124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:50.559134 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:50.559180 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:50.560707 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.561433 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.562524 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.562938 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.563034 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:50.565463 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:50.565466 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:50.565466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:50.565474 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:50.567544 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.568432 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.568433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.568433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.568599 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.568839 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.568840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.568840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.568991 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:50.569069 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:50.569305 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 01:00:50.569399 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 01:00:50.570837 5923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.558964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37202 (local address 127.23.242.60:34743)
0806 01:00:50.559010 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:50.559012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:50.562766 (+ 3754us) server_negotiation.cc:407] Connection header received
0806 01:00:50.562890 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:50.562891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:50.562908 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:50.562946 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:50.563615 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.564316 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.565293 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.565638 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.565683 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:50.567368 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:50.567370 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:50.567370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:50.567378 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:50.569073 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.569488 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.569489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.569489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.569544 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.570122 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.570123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.570124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.570318 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:50.570399 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:50.570700 (+ 301us) server_negotiation.cc:299] Negotiation successful
0806 01:00:50.570776 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 01:00:50.582904 5841 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } attempt: 3
Aug 06 01:00:50 dist-test-slave-dhph krb5kdc[5681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978050, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.61@KRBTEST.COM for kudu/127.23.242.60@KRBTEST.COM
I20260806 01:00:50.592396 5844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.583377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:34743 (local address 127.0.0.1:37218)
0806 01:00:50.583528 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:50.583542 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:50.583596 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:50.583822 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:50.583823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:50.583976 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:00:50.584019 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:50.584021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.585052 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.585053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:50.585784 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:50.585787 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.586537 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.586538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:50.586753 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:50.587269 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:00:50.587276 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:00:50.588569 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:00:50.591156 (+ 2587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:50.591157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:50.591158 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:50.591357 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:50.591604 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:50.591605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:50.591606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:50.591676 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:50.591973 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:00:50.591974 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:00:50.592028 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 01:00:50.592163 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 01:00:50.592278 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260806 01:00:50.592410 5923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.583566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37218 (local address 127.23.242.60:34743)
0806 01:00:50.583682 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:50.583684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:50.583690 (+ 6us) server_negotiation.cc:407] Connection header received
0806 01:00:50.583709 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:50.583710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:50.583720 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:50.583768 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:50.584107 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.584940 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.586000 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.586394 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.586501 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:50.588848 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:50.588849 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:50.588849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:50.588857 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:50.591039 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.591459 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.591460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.591461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.591526 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.591767 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.591768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.591768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.591896 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:50.591981 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:50.592236 (+ 255us) server_negotiation.cc:299] Negotiation successful
0806 01:00:50.592319 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 01:00:50.594264 5841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d6dd9a35e940e092e44e0887409a1d: Bootstrap starting.
I20260806 01:00:50.595247 5841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0d6dd9a35e940e092e44e0887409a1d: Neither blocks nor log segments found. Creating new log.
I20260806 01:00:50.595587 5841 log.cc:824] T 00000000000000000000000000000000 P c0d6dd9a35e940e092e44e0887409a1d: Log is configured to *not* fsync() on all Append() calls
I20260806 01:00:50.599814 5841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d6dd9a35e940e092e44e0887409a1d: No bootstrap required, opened a new log
I20260806 01:00:50.602239 5841 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0d6dd9a35e940e092e44e0887409a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ee7af2dc21485e89cf198997caa562" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44525 } } peers { permanent_uuid: "c0d6dd9a35e940e092e44e0887409a1d" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45479 } } peers { permanent_uuid: "ddcc94a2f600434186509efae6d4dcf9" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } }
I20260806 01:00:50.602417 5841 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0d6dd9a35e940e092e44e0887409a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:00:50.602456 5841 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0d6dd9a35e940e092e44e0887409a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0d6dd9a35e940e092e44e0887409a1d, State: Initialized, Role: FOLLOWER
I20260806 01:00:50.602653 5841 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0d6dd9a35e940e092e44e0887409a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ee7af2dc21485e89cf198997caa562" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44525 } } peers { permanent_uuid: "c0d6dd9a35e940e092e44e0887409a1d" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45479 } } peers { permanent_uuid: "ddcc94a2f600434186509efae6d4dcf9" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } }
I20260806 01:00:50.603047 5841 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0d6dd9a35e940e092e44e0887409a1d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:00:50.603085 5947 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d6dd9a35e940e092e44e0887409a1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ee7af2dc21485e89cf198997caa562" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44525 } } peers { permanent_uuid: "c0d6dd9a35e940e092e44e0887409a1d" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45479 } } peers { permanent_uuid: "ddcc94a2f600434186509efae6d4dcf9" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } } }
I20260806 01:00:50.603149 5947 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d6dd9a35e940e092e44e0887409a1d [sys.catalog]: This master's current role is: FOLLOWER
I20260806 01:00:50.603286 5764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } attempt: 3
W20260806 01:00:50.604946 5958 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0d6dd9a35e940e092e44e0887409a1d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 01:00:50.605015 5958 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 01:00:50.605206 5771 master_runner.cc:186] Error getting master registration for 127.23.242.62:44525: OK, instance_id { permanent_uuid: "c9ee7af2dc21485e89cf198997caa562" instance_seqno: 1785978050154567 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 01:00:50.607793 5771 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 06 01:00:50 dist-test-slave-dhph krb5kdc[5681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978050, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for kudu/127.23.242.60@KRBTEST.COM
I20260806 01:00:50.616653 5923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.604150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37228 (local address 127.23.242.60:34743)
0806 01:00:50.604317 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:50.604319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:50.604328 (+ 9us) server_negotiation.cc:407] Connection header received
0806 01:00:50.604356 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:50.604357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:50.604367 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:50.604835 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:50.604872 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.605702 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.606920 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.607217 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.610689 (+ 3472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:50.610786 (+ 97us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:50.610787 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:50.610787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:50.610793 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:50.615022 (+ 4229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.615523 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.615524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.615525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.615599 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:50.615905 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:50.615907 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:50.615907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:50.616095 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:50.616175 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:50.616456 (+ 281us) server_negotiation.cc:299] Negotiation successful
0806 01:00:50.616558 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260806 01:00:50.616653 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:50.603906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:34743 (local address 127.0.0.1:37228)
0806 01:00:50.603968 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:50.603979 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:50.604022 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:50.604458 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:50.604459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:50.604670 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:00:50.604728 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:50.604730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.605819 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.605821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:50.606668 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:50.606670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:50.607320 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:50.607322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:50.607389 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:50.608668 (+ 1279us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:00:50.608679 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:00:50.610682 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:00:50.615153 (+ 4471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:50.615155 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:50.615155 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:50.615426 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:50.615753 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:50.615754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:50.615754 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:50.615825 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:50.616262 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:00:50.616264 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:00:50.616325 (+ 61us) client_negotiation.cc:769] Sending connection context
0806 01:00:50.616411 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 01:00:50.616532 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 01:00:50.619920 5764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9ee7af2dc21485e89cf198997caa562: Bootstrap starting.
I20260806 01:00:50.620694 5764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9ee7af2dc21485e89cf198997caa562: Neither blocks nor log segments found. Creating new log.
I20260806 01:00:50.621018 5764 log.cc:824] T 00000000000000000000000000000000 P c9ee7af2dc21485e89cf198997caa562: Log is configured to *not* fsync() on all Append() calls
I20260806 01:00:50.621910 5764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9ee7af2dc21485e89cf198997caa562: No bootstrap required, opened a new log
I20260806 01:00:50.623984 5764 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9ee7af2dc21485e89cf198997caa562 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ee7af2dc21485e89cf198997caa562" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44525 } } peers { permanent_uuid: "c0d6dd9a35e940e092e44e0887409a1d" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45479 } } peers { permanent_uuid: "ddcc94a2f600434186509efae6d4dcf9" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } }
I20260806 01:00:50.624133 5764 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9ee7af2dc21485e89cf198997caa562 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:00:50.624168 5764 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9ee7af2dc21485e89cf198997caa562 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9ee7af2dc21485e89cf198997caa562, State: Initialized, Role: FOLLOWER
I20260806 01:00:50.624317 5764 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9ee7af2dc21485e89cf198997caa562 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ee7af2dc21485e89cf198997caa562" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44525 } } peers { permanent_uuid: "c0d6dd9a35e940e092e44e0887409a1d" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45479 } } peers { permanent_uuid: "ddcc94a2f600434186509efae6d4dcf9" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } }
I20260806 01:00:50.624658 5764 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9ee7af2dc21485e89cf198997caa562 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:00:50.624636 5961 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9ee7af2dc21485e89cf198997caa562 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ee7af2dc21485e89cf198997caa562" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44525 } } peers { permanent_uuid: "c0d6dd9a35e940e092e44e0887409a1d" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45479 } } peers { permanent_uuid: "ddcc94a2f600434186509efae6d4dcf9" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } } }
I20260806 01:00:50.624846 5961 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9ee7af2dc21485e89cf198997caa562 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 01:00:50.626549 5972 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c9ee7af2dc21485e89cf198997caa562: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 01:00:50.626688 5972 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 01:00:50.630385 5694 master_runner.cc:263] No leader master found from master c9ee7af2dc21485e89cf198997caa562
I20260806 01:00:50.630470 5694 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 01:00:50.652339 5716 tablet.cc:2369] T 00000000000000000000000000000000 P c9ee7af2dc21485e89cf198997caa562: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 01:00:50.696707 5926 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 01:00:50.696862 5926 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ee7af2dc21485e89cf198997caa562" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44525 } } peers { permanent_uuid: "c0d6dd9a35e940e092e44e0887409a1d" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45479 } } peers { permanent_uuid: "ddcc94a2f600434186509efae6d4dcf9" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } }
I20260806 01:00:50.697116 5926 leader_election.cc:288] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9ee7af2dc21485e89cf198997caa562 (127.23.242.62:44525), c0d6dd9a35e940e092e44e0887409a1d (127.23.242.61:45479)
I20260806 01:00:50.697700 5737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ddcc94a2f600434186509efae6d4dcf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ee7af2dc21485e89cf198997caa562" is_pre_election: true
I20260806 01:00:50.697882 5737 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c9ee7af2dc21485e89cf198997caa562 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddcc94a2f600434186509efae6d4dcf9 in term 0.
I20260806 01:00:50.698201 5866 leader_election.cc:302] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9ee7af2dc21485e89cf198997caa562, ddcc94a2f600434186509efae6d4dcf9; no voters:
I20260806 01:00:50.698833 5926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 01:00:50.698925 5926 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 01:00:50.698957 5926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:00:50.700011 5926 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ee7af2dc21485e89cf198997caa562" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44525 } } peers { permanent_uuid: "c0d6dd9a35e940e092e44e0887409a1d" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45479 } } peers { permanent_uuid: "ddcc94a2f600434186509efae6d4dcf9" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } }
I20260806 01:00:50.700162 5926 leader_election.cc:288] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [CANDIDATE]: Term 1 election: Requested vote from peers c9ee7af2dc21485e89cf198997caa562 (127.23.242.62:44525), c0d6dd9a35e940e092e44e0887409a1d (127.23.242.61:45479)
I20260806 01:00:50.700558 5813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ddcc94a2f600434186509efae6d4dcf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0d6dd9a35e940e092e44e0887409a1d"
I20260806 01:00:50.700706 5813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0d6dd9a35e940e092e44e0887409a1d [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:00:50.701639 5813 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c0d6dd9a35e940e092e44e0887409a1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddcc94a2f600434186509efae6d4dcf9 in term 1.
I20260806 01:00:50.701902 5814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ddcc94a2f600434186509efae6d4dcf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0d6dd9a35e940e092e44e0887409a1d" is_pre_election: true
I20260806 01:00:50.701990 5814 raft_consensus.cc:2362] T 00000000000000000000000000000000 P c0d6dd9a35e940e092e44e0887409a1d [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ddcc94a2f600434186509efae6d4dcf9 in term 1. Re-sending same reply.
I20260806 01:00:50.702272 5737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ddcc94a2f600434186509efae6d4dcf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ee7af2dc21485e89cf198997caa562"
I20260806 01:00:50.702370 5737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9ee7af2dc21485e89cf198997caa562 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:00:50.703347 5737 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c9ee7af2dc21485e89cf198997caa562 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddcc94a2f600434186509efae6d4dcf9 in term 1.
I20260806 01:00:50.703600 5866 leader_election.cc:302] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9ee7af2dc21485e89cf198997caa562, ddcc94a2f600434186509efae6d4dcf9; no voters:
I20260806 01:00:50.705013 5926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:00:50.705202 5926 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [term 1 LEADER]: Becoming Leader. State: Replica: ddcc94a2f600434186509efae6d4dcf9, State: Running, Role: LEADER
I20260806 01:00:50.705307 5926 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ee7af2dc21485e89cf198997caa562" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44525 } } peers { permanent_uuid: "c0d6dd9a35e940e092e44e0887409a1d" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45479 } } peers { permanent_uuid: "ddcc94a2f600434186509efae6d4dcf9" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } }
I20260806 01:00:50.705677 5926 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddcc94a2f600434186509efae6d4dcf9. Latest consensus state: current_term: 1 leader_uuid: "ddcc94a2f600434186509efae6d4dcf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ee7af2dc21485e89cf198997caa562" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44525 } } peers { permanent_uuid: "c0d6dd9a35e940e092e44e0887409a1d" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45479 } } peers { permanent_uuid: "ddcc94a2f600434186509efae6d4dcf9" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } } }
I20260806 01:00:50.705788 5926 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [sys.catalog]: This master's current role is: LEADER
I20260806 01:00:50.706152 5976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:00:50.706377 5976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:00:50.708400 5814 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c0d6dd9a35e940e092e44e0887409a1d [term 1 FOLLOWER]: Refusing update from remote peer ddcc94a2f600434186509efae6d4dcf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:00:50.708709 5926 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0d6dd9a35e940e092e44e0887409a1d" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:00:50.710105 5947 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d6dd9a35e940e092e44e0887409a1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddcc94a2f600434186509efae6d4dcf9. Latest consensus state: current_term: 1 leader_uuid: "ddcc94a2f600434186509efae6d4dcf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ee7af2dc21485e89cf198997caa562" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44525 } } peers { permanent_uuid: "c0d6dd9a35e940e092e44e0887409a1d" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45479 } } peers { permanent_uuid: "ddcc94a2f600434186509efae6d4dcf9" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } } }
I20260806 01:00:50.710207 5947 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d6dd9a35e940e092e44e0887409a1d [sys.catalog]: This master's current role is: FOLLOWER
I20260806 01:00:50.710647 5737 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c9ee7af2dc21485e89cf198997caa562 [term 1 FOLLOWER]: Refusing update from remote peer ddcc94a2f600434186509efae6d4dcf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:00:50.710951 5926 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9ee7af2dc21485e89cf198997caa562" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:00:50.712196 5961 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9ee7af2dc21485e89cf198997caa562 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddcc94a2f600434186509efae6d4dcf9. Latest consensus state: current_term: 1 leader_uuid: "ddcc94a2f600434186509efae6d4dcf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ee7af2dc21485e89cf198997caa562" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44525 } } peers { permanent_uuid: "c0d6dd9a35e940e092e44e0887409a1d" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45479 } } peers { permanent_uuid: "ddcc94a2f600434186509efae6d4dcf9" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } } }
I20260806 01:00:50.712306 5961 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9ee7af2dc21485e89cf198997caa562 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 01:00:50.713949 5974 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ddcc94a2f600434186509efae6d4dcf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ee7af2dc21485e89cf198997caa562" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44525 } } peers { permanent_uuid: "c0d6dd9a35e940e092e44e0887409a1d" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45479 } } peers { permanent_uuid: "ddcc94a2f600434186509efae6d4dcf9" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } } }
I20260806 01:00:50.714073 5974 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [sys.catalog]: This master's current role is: LEADER
I20260806 01:00:50.714488 5961 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9ee7af2dc21485e89cf198997caa562 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ddcc94a2f600434186509efae6d4dcf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ee7af2dc21485e89cf198997caa562" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44525 } } peers { permanent_uuid: "c0d6dd9a35e940e092e44e0887409a1d" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45479 } } peers { permanent_uuid: "ddcc94a2f600434186509efae6d4dcf9" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } } }
I20260806 01:00:50.714599 5961 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9ee7af2dc21485e89cf198997caa562 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 01:00:50.714830 5978 mvcc.cc:205] Tried to move back new op lower bound from 7315366095699214336 to 7315366095689482240. Current Snapshot: MvccSnapshot[applied={T|T < 7315366095699214336}]
I20260806 01:00:50.715272 5981 mvcc.cc:205] Tried to move back new op lower bound from 7315366095699214336 to 7315366095689482240. Current Snapshot: MvccSnapshot[applied={T|T < 7315366095699214336}]
I20260806 01:00:50.715749 5926 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ddcc94a2f600434186509efae6d4dcf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ee7af2dc21485e89cf198997caa562" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44525 } } peers { permanent_uuid: "c0d6dd9a35e940e092e44e0887409a1d" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45479 } } peers { permanent_uuid: "ddcc94a2f600434186509efae6d4dcf9" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } } }
I20260806 01:00:50.715840 5926 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9 [sys.catalog]: This master's current role is: LEADER
I20260806 01:00:50.716158 5947 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d6dd9a35e940e092e44e0887409a1d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ddcc94a2f600434186509efae6d4dcf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9ee7af2dc21485e89cf198997caa562" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44525 } } peers { permanent_uuid: "c0d6dd9a35e940e092e44e0887409a1d" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45479 } } peers { permanent_uuid: "ddcc94a2f600434186509efae6d4dcf9" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 34743 } } }
I20260806 01:00:50.716250 5947 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d6dd9a35e940e092e44e0887409a1d [sys.catalog]: This master's current role is: FOLLOWER
I20260806 01:00:50.716833 5979 mvcc.cc:205] Tried to move back new op lower bound from 7315366095699214336 to 7315366095689482240. Current Snapshot: MvccSnapshot[applied={T|T < 7315366095699214336}]
I20260806 01:00:50.717329 5976 catalog_manager.cc:1347] Generated new cluster ID: 5873f83322234ebc9de6c3fa87ffb035
I20260806 01:00:50.717378 5976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:00:50.729597 5976 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:00:50.730351 5976 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:00:50.741869 5976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddcc94a2f600434186509efae6d4dcf9: Generated new TSK 0
I20260806 01:00:50.742102 5976 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 01:00:51.605501 5958 catalog_manager.cc:1259] Loaded cluster ID: 5873f83322234ebc9de6c3fa87ffb035
I20260806 01:00:51.605593 5958 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c0d6dd9a35e940e092e44e0887409a1d: loading cluster ID for follower catalog manager: success
I20260806 01:00:51.606495 5958 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c0d6dd9a35e940e092e44e0887409a1d: acquiring CA information for follower catalog manager: success
I20260806 01:00:51.606873 5958 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c0d6dd9a35e940e092e44e0887409a1d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 01:00:51.627291 5972 catalog_manager.cc:1259] Loaded cluster ID: 5873f83322234ebc9de6c3fa87ffb035
I20260806 01:00:51.627383 5972 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c9ee7af2dc21485e89cf198997caa562: loading cluster ID for follower catalog manager: success
I20260806 01:00:51.628340 5972 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c9ee7af2dc21485e89cf198997caa562: acquiring CA information for follower catalog manager: success
I20260806 01:00:51.628692 5972 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c9ee7af2dc21485e89cf198997caa562: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 01:00:51.793673 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 5694
I20260806 01:00:51.801267 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 5771
I20260806 01:00:51.808894 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 5848
2026-08-06T01:00:51Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3223 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 01:00:51 dist-test-slave-dhph krb5kdc[5994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 01:00:51 dist-test-slave-dhph krb5kdc[5994](info): set up 2 sockets
Aug 06 01:00:51 dist-test-slave-dhph krb5kdc[5994](info): commencing operation
krb5kdc: starting...
W20260806 01:00:53.026551 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.194s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 01:00:53 dist-test-slave-dhph krb5kdc[5994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T01:00:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:00:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 01:00:53.174573 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:39229
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:42679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:39229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:00:53.266654 6007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:00:53.266943 6007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:00:53.266976 6007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:00:53.268810 6007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:00:53.268865 6007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:00:53.268882 6007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:00:53.268898 6007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:00:53.268913 6007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:00:53.270792 6007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:42679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:39229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:39229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:00:53.271076 6007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:00:53.271365 6007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:00:53.277565 6013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:53.277593 6012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:53.277630 6015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:53.277808 6007 server_base.cc:1034] running on GCE node
I20260806 01:00:53.278216 6007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:00:53.278558 6007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:00:53.279790 6007 hybrid_clock.cc:648] HybridClock initialized: now 1785978053279716 us; error 89 us; skew 500 ppm
Aug 06 01:00:53 dist-test-slave-dhph krb5kdc[5994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978053, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:00:53.282792 6007 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 01:00:53.283347 6007 webserver.cc:460] Webserver started at http://127.23.242.62:34045/ using document root <none> and password file <none>
I20260806 01:00:53.283602 6007 fs_manager.cc:359] Metadata directory not provided
I20260806 01:00:53.283655 6007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:00:53.283761 6007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:00:53.284924 6007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "138f2c5296c248e79b78e31afc885945"
format_stamp: "Formatted at 2026-08-06 01:00:53 on dist-test-slave-dhph"
server_key: "25213175971a839945e0761cfee91a46"
server_key_iv: "8f41aa483fadfc4e0ffde2683e54927d"
server_key_version: "encryptionkey@0"
I20260806 01:00:53.285274 6007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "138f2c5296c248e79b78e31afc885945"
format_stamp: "Formatted at 2026-08-06 01:00:53 on dist-test-slave-dhph"
server_key: "25213175971a839945e0761cfee91a46"
server_key_iv: "8f41aa483fadfc4e0ffde2683e54927d"
server_key_version: "encryptionkey@0"
I20260806 01:00:53.287539 6007 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260806 01:00:53.288899 6022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:53.289234 6007 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:00:53.289366 6007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "138f2c5296c248e79b78e31afc885945"
format_stamp: "Formatted at 2026-08-06 01:00:53 on dist-test-slave-dhph"
server_key: "25213175971a839945e0761cfee91a46"
server_key_iv: "8f41aa483fadfc4e0ffde2683e54927d"
server_key_version: "encryptionkey@0"
I20260806 01:00:53.289456 6007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:00:53.305557 6007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:00:53.308270 6007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:00:53.308476 6007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:00:53.313251 6007 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:39229
I20260806 01:00:53.313284 6076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:39229 every 8 connection(s)
I20260806 01:00:53.313649 6007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:00:53.314652 6077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:00:53.318928 6077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 138f2c5296c248e79b78e31afc885945: Bootstrap starting.
I20260806 01:00:53.319828 6077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 138f2c5296c248e79b78e31afc885945: Neither blocks nor log segments found. Creating new log.
I20260806 01:00:53.320227 6077 log.cc:824] T 00000000000000000000000000000000 P 138f2c5296c248e79b78e31afc885945: Log is configured to *not* fsync() on all Append() calls
I20260806 01:00:53.321431 6077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 138f2c5296c248e79b78e31afc885945: No bootstrap required, opened a new log
I20260806 01:00:53.321857 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 6007
I20260806 01:00:53.322005 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:00:53.322201 24520 external_mini_cluster.cc:1428] Setting key 0f0b1b5fbd30a9b36fca5c36d4c3306c
I20260806 01:00:53.324306 6077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 138f2c5296c248e79b78e31afc885945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138f2c5296c248e79b78e31afc885945" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39229 } }
I20260806 01:00:53.324541 6077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 138f2c5296c248e79b78e31afc885945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:00:53.324595 6077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 138f2c5296c248e79b78e31afc885945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138f2c5296c248e79b78e31afc885945, State: Initialized, Role: FOLLOWER
I20260806 01:00:53.324779 6077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 138f2c5296c248e79b78e31afc885945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138f2c5296c248e79b78e31afc885945" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39229 } }
I20260806 01:00:53.324898 6077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 138f2c5296c248e79b78e31afc885945 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:00:53.324946 6077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 138f2c5296c248e79b78e31afc885945 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:00:53.324999 6077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 138f2c5296c248e79b78e31afc885945 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:00:53.326040 6077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 138f2c5296c248e79b78e31afc885945 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138f2c5296c248e79b78e31afc885945" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39229 } }
I20260806 01:00:53.326174 6077 leader_election.cc:302] T 00000000000000000000000000000000 P 138f2c5296c248e79b78e31afc885945 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 138f2c5296c248e79b78e31afc885945; no voters:
I20260806 01:00:53.326388 6077 leader_election.cc:288] T 00000000000000000000000000000000 P 138f2c5296c248e79b78e31afc885945 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:00:53.326443 6082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 138f2c5296c248e79b78e31afc885945 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:00:53.326653 6082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 138f2c5296c248e79b78e31afc885945 [term 1 LEADER]: Becoming Leader. State: Replica: 138f2c5296c248e79b78e31afc885945, State: Running, Role: LEADER
I20260806 01:00:53.326720 6077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 138f2c5296c248e79b78e31afc885945 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:00:53.326752 6082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 138f2c5296c248e79b78e31afc885945 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138f2c5296c248e79b78e31afc885945" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39229 } }
I20260806 01:00:53.328405 6084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 138f2c5296c248e79b78e31afc885945 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "138f2c5296c248e79b78e31afc885945" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138f2c5296c248e79b78e31afc885945" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39229 } } }
I20260806 01:00:53.328405 6083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 138f2c5296c248e79b78e31afc885945 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 138f2c5296c248e79b78e31afc885945. Latest consensus state: current_term: 1 leader_uuid: "138f2c5296c248e79b78e31afc885945" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138f2c5296c248e79b78e31afc885945" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39229 } } }
I20260806 01:00:53.328711 6084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 138f2c5296c248e79b78e31afc885945 [sys.catalog]: This master's current role is: LEADER
I20260806 01:00:53.328722 6083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 138f2c5296c248e79b78e31afc885945 [sys.catalog]: This master's current role is: LEADER
W20260806 01:00:53.329335 6096 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 138f2c5296c248e79b78e31afc885945: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 01:00:53.329418 6096 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 01:00:53.329514 6097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:00:53.329728 6097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 06 01:00:53 dist-test-slave-dhph krb5kdc[5994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:00:53.332090 6097 catalog_manager.cc:1347] Generated new cluster ID: eac947b112904b928ee1a7b66d1e7b75
I20260806 01:00:53.332183 6097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:00:53.336565 6081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:53.323039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39728 (local address 127.23.242.62:39229)
0806 01:00:53.323309 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:53.323317 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:53.323353 (+ 36us) server_negotiation.cc:407] Connection header received
0806 01:00:53.324324 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:53.324333 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:53.324443 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:53.324631 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:53.325523 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:53.326892 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:53.327916 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:53.328649 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:53.328770 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:53.332073 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:53.332077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:53.332077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:53.332089 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:53.334278 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:53.334767 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:53.334771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:53.334772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:53.334922 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:53.335167 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:53.335168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:53.335169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:53.335525 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:53.336115 (+ 590us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:53.336204 (+ 89us) server_negotiation.cc:299] Negotiation successful
0806 01:00:53.336350 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":146,"threads_started":1}
I20260806 01:00:53.337618 24520 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 01:00:53.347419 6097 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:00:53.347990 6097 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:00:53.353255 6097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 138f2c5296c248e79b78e31afc885945: Generated new TSK 0
I20260806 01:00:53.353583 6097 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 01:00:53.392515 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 6007
2026-08-06T01:00:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1581 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 01:00:53 dist-test-slave-dhph krb5kdc[6104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 01:00:53 dist-test-slave-dhph krb5kdc[6104](info): set up 2 sockets
Aug 06 01:00:53 dist-test-slave-dhph krb5kdc[6104](info): commencing operation
krb5kdc: starting...
W20260806 01:00:54.590216 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 01:00:54 dist-test-slave-dhph krb5kdc[6104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T01:00:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:00:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 01:00:54.743340 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:43009
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:41023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:43009
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:00:54.839843 6117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:00:54.840081 6117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:00:54.840109 6117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:00:54.841356 6117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:00:54.841398 6117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:00:54.841413 6117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:00:54.841426 6117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:00:54.841439 6117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:00:54.843058 6117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:43009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:43009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:00:54.843295 6117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:00:54.843489 6117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:00:54.849964 6123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:54.849970 6124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:54.849946 6126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:54.851384 6117 server_base.cc:1034] running on GCE node
I20260806 01:00:54.851814 6117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:00:54.852242 6117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:00:54.853466 6117 hybrid_clock.cc:648] HybridClock initialized: now 1785978054853488 us; error 80 us; skew 500 ppm
Aug 06 01:00:54 dist-test-slave-dhph krb5kdc[6104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978054, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:00:54.857213 6117 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 01:00:54.857846 6117 webserver.cc:460] Webserver started at http://127.23.242.62:45829/ using document root <none> and password file <none>
I20260806 01:00:54.858140 6117 fs_manager.cc:359] Metadata directory not provided
I20260806 01:00:54.858222 6117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:00:54.858395 6117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:00:54.859717 6117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "ef9c497be1c44b0e80cdbbb063aac875"
format_stamp: "Formatted at 2026-08-06 01:00:54 on dist-test-slave-dhph"
server_key: "ada6fa0f6ac4f6c732770b82dc5d2a25"
server_key_iv: "137db35b00a0673397254544ae372e64"
server_key_version: "encryptionkey@0"
I20260806 01:00:54.860155 6117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "ef9c497be1c44b0e80cdbbb063aac875"
format_stamp: "Formatted at 2026-08-06 01:00:54 on dist-test-slave-dhph"
server_key: "ada6fa0f6ac4f6c732770b82dc5d2a25"
server_key_iv: "137db35b00a0673397254544ae372e64"
server_key_version: "encryptionkey@0"
I20260806 01:00:54.862028 6117 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260806 01:00:54.863142 6133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:54.863421 6117 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:00:54.863520 6117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "ef9c497be1c44b0e80cdbbb063aac875"
format_stamp: "Formatted at 2026-08-06 01:00:54 on dist-test-slave-dhph"
server_key: "ada6fa0f6ac4f6c732770b82dc5d2a25"
server_key_iv: "137db35b00a0673397254544ae372e64"
server_key_version: "encryptionkey@0"
I20260806 01:00:54.863574 6117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:00:54.893074 6117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:00:54.895798 6117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:00:54.895941 6117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:00:54.901621 6117 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:43009
I20260806 01:00:54.901901 6187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:43009 every 8 connection(s)
I20260806 01:00:54.902092 6117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:00:54.903365 6188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:00:54.906018 6188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875: Bootstrap starting.
I20260806 01:00:54.906697 6188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875: Neither blocks nor log segments found. Creating new log.
I20260806 01:00:54.906978 6188 log.cc:824] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875: Log is configured to *not* fsync() on all Append() calls
I20260806 01:00:54.907791 6188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875: No bootstrap required, opened a new log
I20260806 01:00:54.909602 6188 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9c497be1c44b0e80cdbbb063aac875" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43009 } }
I20260806 01:00:54.909785 6188 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:00:54.909811 6188 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef9c497be1c44b0e80cdbbb063aac875, State: Initialized, Role: FOLLOWER
I20260806 01:00:54.909970 6188 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9c497be1c44b0e80cdbbb063aac875" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43009 } }
I20260806 01:00:54.910063 6188 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:00:54.910090 6188 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:00:54.910123 6188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:00:54.911093 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 6117
I20260806 01:00:54.911525 6188 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9c497be1c44b0e80cdbbb063aac875" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43009 } }
I20260806 01:00:54.911598 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:00:54.911692 6188 leader_election.cc:302] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef9c497be1c44b0e80cdbbb063aac875; no voters:
I20260806 01:00:54.911880 24520 external_mini_cluster.cc:1428] Setting key 878cd02540eedced185d21a8f677000f
I20260806 01:00:54.911911 6188 leader_election.cc:288] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:00:54.911976 6191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:00:54.912164 6191 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [term 1 LEADER]: Becoming Leader. State: Replica: ef9c497be1c44b0e80cdbbb063aac875, State: Running, Role: LEADER
I20260806 01:00:54.912235 6188 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:00:54.912271 6191 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9c497be1c44b0e80cdbbb063aac875" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43009 } }
I20260806 01:00:54.912844 6192 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef9c497be1c44b0e80cdbbb063aac875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9c497be1c44b0e80cdbbb063aac875" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43009 } } }
I20260806 01:00:54.912940 6192 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [sys.catalog]: This master's current role is: LEADER
I20260806 01:00:54.913101 6193 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef9c497be1c44b0e80cdbbb063aac875. Latest consensus state: current_term: 1 leader_uuid: "ef9c497be1c44b0e80cdbbb063aac875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9c497be1c44b0e80cdbbb063aac875" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43009 } } }
I20260806 01:00:54.913166 6193 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [sys.catalog]: This master's current role is: LEADER
I20260806 01:00:54.913458 6199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:00:54.914068 6199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:00:54.916106 6199 catalog_manager.cc:1347] Generated new cluster ID: 32b0a8237d7e4979ae20796c586c2c7f
I20260806 01:00:54.916186 6199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 01:00:54 dist-test-slave-dhph krb5kdc[6104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:00:54.926901 6196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:54.912667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39036 (local address 127.23.242.62:43009)
0806 01:00:54.912840 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:54.912851 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:54.912890 (+ 39us) server_negotiation.cc:407] Connection header received
0806 01:00:54.913029 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:54.913037 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:54.913123 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:54.914309 (+ 1186us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:54.914423 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:54.915970 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:54.917204 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:54.917923 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:54.918031 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:54.921657 (+ 3626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:54.921663 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:54.921663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:54.921680 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:54.924793 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:54.925278 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:54.925282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:54.925282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:54.925425 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:54.925655 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:54.925655 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:54.925656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:54.926035 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:54.926286 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:54.926381 (+ 95us) server_negotiation.cc:299] Negotiation successful
0806 01:00:54.926571 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260806 01:00:54.931293 6199 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:00:54.932297 6199 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:00:54.940760 6199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875: Generated new TSK 0
I20260806 01:00:54.941008 6199 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.242.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
I20260806 01:00:54.990826 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:0
--local_ip_for_outbound_sockets=127.23.242.1
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:43009
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:41023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:00:55.099525 6214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:00:55.099790 6214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:00:55.099820 6214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:00:55.101250 6214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:00:55.101310 6214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:00:55.101342 6214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 01:00:55.103124 6214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:43009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:00:55.103431 6214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:00:55.103734 6214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:00:55.110544 6219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:55.111459 6220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:55.111567 6222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:55.111635 6214 server_base.cc:1034] running on GCE node
I20260806 01:00:55.112059 6214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:00:55.112877 6214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:00:55.114363 6214 hybrid_clock.cc:648] HybridClock initialized: now 1785978055114333 us; error 59 us; skew 500 ppm
Aug 06 01:00:55 dist-test-slave-dhph krb5kdc[6104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978055, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:00:55.122749 6214 init.cc:377] Logged in from keytab as kudu/127.23.242.1@KRBTEST.COM (short username kudu)
I20260806 01:00:55.123338 6214 webserver.cc:460] Webserver started at http://127.23.242.1:35431/ using document root <none> and password file <none>
I20260806 01:00:55.123620 6214 fs_manager.cc:359] Metadata directory not provided
I20260806 01:00:55.123703 6214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:00:55.123883 6214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:00:55.125155 6214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/data/instance:
uuid: "9f248ea2790943f2b843ad446336501e"
format_stamp: "Formatted at 2026-08-06 01:00:55 on dist-test-slave-dhph"
server_key: "2a2d20195556cab78e89075edfc29feb"
server_key_iv: "3d364d3cd187e0345f3d48a77cc70c9b"
server_key_version: "encryptionkey@0"
I20260806 01:00:55.125588 6214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance:
uuid: "9f248ea2790943f2b843ad446336501e"
format_stamp: "Formatted at 2026-08-06 01:00:55 on dist-test-slave-dhph"
server_key: "2a2d20195556cab78e89075edfc29feb"
server_key_iv: "3d364d3cd187e0345f3d48a77cc70c9b"
server_key_version: "encryptionkey@0"
I20260806 01:00:55.128054 6214 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260806 01:00:55.129223 6229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:55.129573 6214 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260806 01:00:55.129880 6214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "9f248ea2790943f2b843ad446336501e"
format_stamp: "Formatted at 2026-08-06 01:00:55 on dist-test-slave-dhph"
server_key: "2a2d20195556cab78e89075edfc29feb"
server_key_iv: "3d364d3cd187e0345f3d48a77cc70c9b"
server_key_version: "encryptionkey@0"
I20260806 01:00:55.129987 6214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:00:55.156091 6214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:00:55.159327 6214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:00:55.159544 6214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:00:55.159871 6214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:00:55.160863 6214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:00:55.160944 6214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:55.160997 6214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:00:55.161031 6214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:55.171950 6214 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:41361
I20260806 01:00:55.172281 6346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:41361 every 8 connection(s)
I20260806 01:00:55.172473 6214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 01:00:55.179221 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 6214
I20260806 01:00:55.179526 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance
I20260806 01:00:55.179989 24520 external_mini_cluster.cc:1428] Setting key 00070a337f7ce09da4a32d74f5e8b5c1
Aug 06 01:00:55 dist-test-slave-dhph krb5kdc[6104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978055, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:00:55.187964 6196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:55.173522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:40343 (local address 127.23.242.62:43009)
0806 01:00:55.173620 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:55.173622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:55.174783 (+ 1161us) server_negotiation.cc:407] Connection header received
0806 01:00:55.175149 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:55.175152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:55.175185 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:55.175306 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:55.176644 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.177375 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.178988 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.179569 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.179706 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:55.183285 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:55.183287 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:55.183287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:55.183297 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:55.185134 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:55.185886 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:55.185888 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:55.185888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:55.185965 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:55.186767 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:55.186944 (+ 177us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:55.186945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:55.187096 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:55.187256 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:55.187589 (+ 333us) server_negotiation.cc:299] Negotiation successful
0806 01:00:55.187853 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 01:00:55.188133 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:55.174017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.1:40343)
0806 01:00:55.174255 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:55.174305 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:55.174917 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:55.175474 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:55.175479 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:55.176167 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:00:55.176412 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:55.176418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.177769 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.177770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:55.178758 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:55.178761 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.179988 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.179992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:55.180217 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:55.180974 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:00:55.180986 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:00:55.183140 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:00:55.185299 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:55.185312 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:55.185315 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:55.185752 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:55.186115 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:55.186116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:55.186116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:55.186291 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:55.187247 (+ 956us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:00:55.187256 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:00:55.187421 (+ 165us) client_negotiation.cc:769] Sending connection context
0806 01:00:55.187542 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 01:00:55.187839 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":95,"threads_started":1}
I20260806 01:00:55.189352 6347 heartbeater.cc:347] Connected to a master server at 127.23.242.62:43009
I20260806 01:00:55.189457 6347 heartbeater.cc:464] Registering TS with master...
I20260806 01:00:55.190018 6347 heartbeater.cc:511] Master 127.23.242.62:43009 requested a full tablet report, sending...
I20260806 01:00:55.191110 6150 ts_manager.cc:194] Registered new tserver with Master: 9f248ea2790943f2b843ad446336501e (127.23.242.1:41361)
I20260806 01:00:55.192250 6150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.1@KRBTEST.COM'} at 127.23.242.1:40343
WARNING: no policy specified for kudu/127.23.242.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
I20260806 01:00:55.224018 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:0
--local_ip_for_outbound_sockets=127.23.242.2
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:43009
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:41023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:00:55.326753 6352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:00:55.326999 6352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:00:55.327032 6352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:00:55.328326 6352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:00:55.328385 6352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:00:55.328418 6352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 01:00:55.330936 6352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:43009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:00:55.331200 6352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:00:55.331425 6352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:00:55.337839 6357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:55.337875 6358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:55.337895 6360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:55.338835 6352 server_base.cc:1034] running on GCE node
I20260806 01:00:55.339216 6352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:00:55.339565 6352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:00:55.340874 6352 hybrid_clock.cc:648] HybridClock initialized: now 1785978055340872 us; error 75 us; skew 500 ppm
Aug 06 01:00:55 dist-test-slave-dhph krb5kdc[6104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978055, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:00:55.344420 6352 init.cc:377] Logged in from keytab as kudu/127.23.242.2@KRBTEST.COM (short username kudu)
I20260806 01:00:55.344951 6352 webserver.cc:460] Webserver started at http://127.23.242.2:36743/ using document root <none> and password file <none>
I20260806 01:00:55.345218 6352 fs_manager.cc:359] Metadata directory not provided
I20260806 01:00:55.345290 6352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:00:55.345444 6352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:00:55.346647 6352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/data/instance:
uuid: "9535377387324589a3599fe08d7eb908"
format_stamp: "Formatted at 2026-08-06 01:00:55 on dist-test-slave-dhph"
server_key: "2162ed441f203abeccede0ce1e71fcf2"
server_key_iv: "87a15761a8ec33827ac3242d36405cf0"
server_key_version: "encryptionkey@0"
I20260806 01:00:55.347194 6352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance:
uuid: "9535377387324589a3599fe08d7eb908"
format_stamp: "Formatted at 2026-08-06 01:00:55 on dist-test-slave-dhph"
server_key: "2162ed441f203abeccede0ce1e71fcf2"
server_key_iv: "87a15761a8ec33827ac3242d36405cf0"
server_key_version: "encryptionkey@0"
I20260806 01:00:55.348850 6352 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 01:00:55.350124 6367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:55.350540 6352 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:00:55.350849 6352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "9535377387324589a3599fe08d7eb908"
format_stamp: "Formatted at 2026-08-06 01:00:55 on dist-test-slave-dhph"
server_key: "2162ed441f203abeccede0ce1e71fcf2"
server_key_iv: "87a15761a8ec33827ac3242d36405cf0"
server_key_version: "encryptionkey@0"
I20260806 01:00:55.350950 6352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:00:55.368362 6352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:00:55.371347 6352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:00:55.371526 6352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:00:55.371850 6352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:00:55.372256 6352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:00:55.372303 6352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:55.372329 6352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:00:55.372344 6352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:55.379350 6352 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:37291
I20260806 01:00:55.379428 6483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:37291 every 8 connection(s)
I20260806 01:00:55.379987 6352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 01:00:55.381577 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 6352
I20260806 01:00:55.381712 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance
I20260806 01:00:55.381939 24520 external_mini_cluster.cc:1428] Setting key 0b48c76e350a1094e6c7cae4345bd6d8
Aug 06 01:00:55 dist-test-slave-dhph krb5kdc[6104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978055, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:00:55.391602 6196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:55.381009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:33029 (local address 127.23.242.62:43009)
0806 01:00:55.381095 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:55.381097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:55.381412 (+ 315us) server_negotiation.cc:407] Connection header received
0806 01:00:55.382053 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:55.382055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:55.382085 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:55.382147 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:55.383455 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.383961 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.384987 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.385182 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.385280 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:55.387872 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:55.387874 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:55.387874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:55.387882 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:55.389686 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:55.390208 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:55.390209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:55.390209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:55.390280 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:55.390576 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:55.390596 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:55.390596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:55.390744 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:55.390835 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:55.391149 (+ 314us) server_negotiation.cc:299] Negotiation successful
0806 01:00:55.391251 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 01:00:55.392163 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:55.380987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.2:33029)
0806 01:00:55.381204 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:55.381279 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:55.381946 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:55.382336 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:55.382342 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:55.383139 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:00:55.383351 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:55.383356 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.384065 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.384066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:55.384891 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:55.384893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.385297 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.385298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:55.385436 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:55.386104 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:00:55.386115 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:00:55.387733 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:00:55.389820 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:55.389831 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:55.389834 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:55.390105 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:55.390372 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:55.390373 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:55.390374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:55.390496 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:55.390825 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:00:55.390833 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:00:55.390996 (+ 163us) client_negotiation.cc:769] Sending connection context
0806 01:00:55.391134 (+ 138us) client_negotiation.cc:240] Negotiation successful
0806 01:00:55.391327 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":114,"threads_started":1}
I20260806 01:00:55.392809 6484 heartbeater.cc:347] Connected to a master server at 127.23.242.62:43009
I20260806 01:00:55.392920 6484 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.242.3@KRBTEST.COM; defaulting to no policy
I20260806 01:00:55.393151 6484 heartbeater.cc:511] Master 127.23.242.62:43009 requested a full tablet report, sending...
I20260806 01:00:55.393599 6150 ts_manager.cc:194] Registered new tserver with Master: 9535377387324589a3599fe08d7eb908 (127.23.242.2:37291)
I20260806 01:00:55.394090 6150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.2@KRBTEST.COM'} at 127.23.242.2:33029
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
I20260806 01:00:55.419629 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:0
--local_ip_for_outbound_sockets=127.23.242.3
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:43009
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:41023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:00:55.523401 6489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:00:55.523669 6489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:00:55.523705 6489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:00:55.525211 6489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:00:55.525285 6489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:00:55.525319 6489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 01:00:55.527801 6489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:43009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:00:55.528093 6489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:00:55.528424 6489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:00:55.535828 6497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:55.535866 6494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:55.535863 6495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:55.536634 6489 server_base.cc:1034] running on GCE node
I20260806 01:00:55.536886 6489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:00:55.537124 6489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:00:55.538316 6489 hybrid_clock.cc:648] HybridClock initialized: now 1785978055538247 us; error 90 us; skew 500 ppm
Aug 06 01:00:55 dist-test-slave-dhph krb5kdc[6104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978055, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:00:55.541203 6489 init.cc:377] Logged in from keytab as kudu/127.23.242.3@KRBTEST.COM (short username kudu)
I20260806 01:00:55.541723 6489 webserver.cc:460] Webserver started at http://127.23.242.3:46735/ using document root <none> and password file <none>
I20260806 01:00:55.541963 6489 fs_manager.cc:359] Metadata directory not provided
I20260806 01:00:55.542034 6489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:00:55.542194 6489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:00:55.543359 6489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/data/instance:
uuid: "f45227c50fdc415c8cb398d7f36bfdeb"
format_stamp: "Formatted at 2026-08-06 01:00:55 on dist-test-slave-dhph"
server_key: "2c9d7693ce1ef7f0243182d01de82540"
server_key_iv: "8a151c8ca02db6200627325899c2b267"
server_key_version: "encryptionkey@0"
I20260806 01:00:55.543742 6489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance:
uuid: "f45227c50fdc415c8cb398d7f36bfdeb"
format_stamp: "Formatted at 2026-08-06 01:00:55 on dist-test-slave-dhph"
server_key: "2c9d7693ce1ef7f0243182d01de82540"
server_key_iv: "8a151c8ca02db6200627325899c2b267"
server_key_version: "encryptionkey@0"
I20260806 01:00:55.545245 6489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 01:00:55.546366 6504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:55.546828 6489 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:00:55.546944 6489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "f45227c50fdc415c8cb398d7f36bfdeb"
format_stamp: "Formatted at 2026-08-06 01:00:55 on dist-test-slave-dhph"
server_key: "2c9d7693ce1ef7f0243182d01de82540"
server_key_iv: "8a151c8ca02db6200627325899c2b267"
server_key_version: "encryptionkey@0"
I20260806 01:00:55.547043 6489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:00:55.560881 6489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:00:55.564047 6489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:00:55.564201 6489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:00:55.564545 6489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:00:55.564929 6489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:00:55.564990 6489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:55.565038 6489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:00:55.565070 6489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:55.573019 6489 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:34067
I20260806 01:00:55.573118 6618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:34067 every 8 connection(s)
I20260806 01:00:55.573526 6489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
I20260806 01:00:55.577742 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 6489
I20260806 01:00:55.577867 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance
I20260806 01:00:55.578115 24520 external_mini_cluster.cc:1428] Setting key 06b75cb9e434ddda0e1ba8fa37c20f6a
Aug 06 01:00:55 dist-test-slave-dhph krb5kdc[6104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978055, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:00:55.586277 6196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:55.574631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:59015 (local address 127.23.242.62:43009)
0806 01:00:55.574717 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:55.574720 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:55.574935 (+ 215us) server_negotiation.cc:407] Connection header received
0806 01:00:55.575580 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:55.575581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:55.575596 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:55.575757 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:55.576875 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.577473 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.579052 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.579287 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.579386 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:55.582638 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:55.582640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:55.582641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:55.582651 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:55.584538 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:55.585091 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:55.585093 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:55.585094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:55.585171 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:55.585470 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:55.585471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:55.585472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:55.585621 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:55.585703 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:55.586049 (+ 346us) server_negotiation.cc:299] Negotiation successful
0806 01:00:55.586149 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 01:00:55.586555 6622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:55.574483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.3:59015)
0806 01:00:55.574791 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:55.574842 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:55.575418 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:55.575851 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:55.575855 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:55.576521 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:00:55.576754 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:55.576759 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.577616 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.577617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:55.578954 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:55.578957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.579402 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.579403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:55.579568 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:55.580579 (+ 1011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:00:55.580589 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:00:55.582458 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:00:55.584677 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:55.584691 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:55.584694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:55.584985 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:55.585252 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:55.585253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:55.585254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:55.585392 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:55.585707 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:00:55.585715 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:00:55.585895 (+ 180us) client_negotiation.cc:769] Sending connection context
0806 01:00:55.586023 (+ 128us) client_negotiation.cc:240] Negotiation successful
0806 01:00:55.586228 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":147,"threads_started":1}
I20260806 01:00:55.587185 6619 heartbeater.cc:347] Connected to a master server at 127.23.242.62:43009
I20260806 01:00:55.587298 6619 heartbeater.cc:464] Registering TS with master...
I20260806 01:00:55.587471 6619 heartbeater.cc:511] Master 127.23.242.62:43009 requested a full tablet report, sending...
I20260806 01:00:55.587903 6150 ts_manager.cc:194] Registered new tserver with Master: f45227c50fdc415c8cb398d7f36bfdeb (127.23.242.3:34067)
I20260806 01:00:55.588305 6150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.3@KRBTEST.COM'} at 127.23.242.3:59015
I20260806 01:00:55.590770 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 01:00:55.600660 6196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:55.592999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39040 (local address 127.23.242.62:43009)
0806 01:00:55.593070 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:55.593072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:55.593083 (+ 11us) server_negotiation.cc:407] Connection header received
0806 01:00:55.593108 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:55.593109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:55.593120 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:55.593201 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:55.593850 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.594390 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.595263 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.595557 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.595662 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:55.597065 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:55.597067 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:55.597067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:55.597077 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:55.598818 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:55.599241 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:55.599243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:55.599243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:55.599314 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:55.599566 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:55.599567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:55.599567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:55.599946 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:55.600272 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:55.600423 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 01:00:55.600527 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 01:00:55.603632 6150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 01:00:55.605466 6150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 01:00:55.615482 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:55.609553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:37291 (local address 127.0.0.1:33164)
0806 01:00:55.610530 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:55.610541 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:55.610915 (+ 374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:55.611354 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:55.611355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:55.611359 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:00:55.611395 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:55.611396 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.612767 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.612768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:55.614030 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:55.614033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.615082 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.615083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:55.615205 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:55.615222 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 01:00:55.615309 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 01:00:55.615371 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"mutex_wait_us":26,"thread_start_us":77,"threads_started":1}
I20260806 01:00:55.616093 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:55.609661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33164 (local address 127.23.242.2:37291)
0806 01:00:55.609869 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:55.609875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:55.610987 (+ 1112us) server_negotiation.cc:407] Connection header received
0806 01:00:55.611230 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:55.611233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:55.611250 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:55.611463 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:00:55.611536 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.612649 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.614136 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.614987 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.615067 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:55.615138 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:55.615336 (+ 198us) server_negotiation.cc:299] Negotiation successful
0806 01:00:55.616005 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":100,"threads_started":1}
I20260806 01:00:55.616143 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:55.609788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:41361 (local address 127.0.0.1:41118)
0806 01:00:55.611260 (+ 1472us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:55.611273 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:55.611343 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:55.611745 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:55.611747 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:55.611749 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:00:55.611815 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:55.611817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.613772 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.613773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:55.614722 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:55.614725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.615842 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.615843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:55.615913 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:55.615918 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 01:00:55.616007 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 01:00:55.616060 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1458,"thread_start_us":360,"threads_started":1}
I20260806 01:00:55.616487 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:55.610560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41118 (local address 127.23.242.1:41361)
0806 01:00:55.611373 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:55.611378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:55.611390 (+ 12us) server_negotiation.cc:407] Connection header received
0806 01:00:55.611442 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:55.611444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:55.611457 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:55.611736 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:00:55.611979 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.613638 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.614816 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.615660 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.615746 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:55.615806 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:55.616349 (+ 543us) server_negotiation.cc:299] Negotiation successful
0806 01:00:55.616400 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":775,"thread_start_us":660,"threads_started":1}
I20260806 01:00:55.616887 6635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:55.609691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49904 (local address 127.23.242.3:34067)
0806 01:00:55.610456 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:55.610461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:55.610815 (+ 354us) server_negotiation.cc:407] Connection header received
0806 01:00:55.610873 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:55.610875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:55.610898 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:55.610981 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:00:55.611291 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.612583 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.615476 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.616420 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.616493 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:55.616545 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:55.616735 (+ 190us) server_negotiation.cc:299] Negotiation successful
0806 01:00:55.616795 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":401,"threads_started":1}
I20260806 01:00:55.617215 6416 tablet_service.cc:1467] Processing CreateTablet for tablet 286e5957de11442aa5b6773d7d5cb8f3 (DEFAULT_TABLE table=test-table [id=3d6c8bf17cb246daaaad48208ceb7a77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:00:55.617489 6279 tablet_service.cc:1467] Processing CreateTablet for tablet 286e5957de11442aa5b6773d7d5cb8f3 (DEFAULT_TABLE table=test-table [id=3d6c8bf17cb246daaaad48208ceb7a77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:00:55.618254 6416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 286e5957de11442aa5b6773d7d5cb8f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:00:55.618573 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 286e5957de11442aa5b6773d7d5cb8f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:00:55.622953 6636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:55.609549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:34067 (local address 127.0.0.1:49904)
0806 01:00:55.610438 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:55.610495 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:55.610701 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:55.611098 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:55.611101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:55.611104 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:00:55.611188 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:55.611190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.612707 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.612708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:55.615332 (+ 2624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:55.615335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.616582 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.616582 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:55.616646 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:55.616650 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 01:00:55.616712 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 01:00:55.622854 (+ 6142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":17,"thread_start_us":94,"threads_started":1}
I20260806 01:00:55.623775 6639 tablet_bootstrap.cc:492] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e: Bootstrap starting.
I20260806 01:00:55.624310 6553 tablet_service.cc:1467] Processing CreateTablet for tablet 286e5957de11442aa5b6773d7d5cb8f3 (DEFAULT_TABLE table=test-table [id=3d6c8bf17cb246daaaad48208ceb7a77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:00:55.624486 6639 tablet_bootstrap.cc:654] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e: Neither blocks nor log segments found. Creating new log.
I20260806 01:00:55.624660 6553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 286e5957de11442aa5b6773d7d5cb8f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:00:55.625180 6639 log.cc:824] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e: Log is configured to *not* fsync() on all Append() calls
I20260806 01:00:55.625646 6638 tablet_bootstrap.cc:492] T 286e5957de11442aa5b6773d7d5cb8f3 P 9535377387324589a3599fe08d7eb908: Bootstrap starting.
I20260806 01:00:55.626279 6639 tablet_bootstrap.cc:492] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e: No bootstrap required, opened a new log
I20260806 01:00:55.626300 6638 tablet_bootstrap.cc:654] T 286e5957de11442aa5b6773d7d5cb8f3 P 9535377387324589a3599fe08d7eb908: Neither blocks nor log segments found. Creating new log.
I20260806 01:00:55.626361 6639 ts_tablet_manager.cc:1397] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 01:00:55.626945 6638 log.cc:824] T 286e5957de11442aa5b6773d7d5cb8f3 P 9535377387324589a3599fe08d7eb908: Log is configured to *not* fsync() on all Append() calls
I20260806 01:00:55.628230 6638 tablet_bootstrap.cc:492] T 286e5957de11442aa5b6773d7d5cb8f3 P 9535377387324589a3599fe08d7eb908: No bootstrap required, opened a new log
I20260806 01:00:55.628331 6638 ts_tablet_manager.cc:1397] T 286e5957de11442aa5b6773d7d5cb8f3 P 9535377387324589a3599fe08d7eb908: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 01:00:55.628561 6642 tablet_bootstrap.cc:492] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb: Bootstrap starting.
I20260806 01:00:55.628614 6639 raft_consensus.cc:348] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } }
I20260806 01:00:55.628849 6639 raft_consensus.cc:374] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:00:55.628896 6639 raft_consensus.cc:729] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f248ea2790943f2b843ad446336501e, State: Initialized, Role: FOLLOWER
I20260806 01:00:55.629128 6639 consensus_queue.cc:260] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } }
I20260806 01:00:55.629333 6642 tablet_bootstrap.cc:654] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb: Neither blocks nor log segments found. Creating new log.
I20260806 01:00:55.629415 6639 ts_tablet_manager.cc:1428] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 01:00:55.629544 6347 heartbeater.cc:503] Master 127.23.242.62:43009 was elected leader, sending a full tablet report...
I20260806 01:00:55.630031 6642 log.cc:824] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb: Log is configured to *not* fsync() on all Append() calls
I20260806 01:00:55.630966 6642 tablet_bootstrap.cc:492] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb: No bootstrap required, opened a new log
I20260806 01:00:55.631044 6642 ts_tablet_manager.cc:1397] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 01:00:55.630941 6638 raft_consensus.cc:348] T 286e5957de11442aa5b6773d7d5cb8f3 P 9535377387324589a3599fe08d7eb908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } }
I20260806 01:00:55.631160 6638 raft_consensus.cc:374] T 286e5957de11442aa5b6773d7d5cb8f3 P 9535377387324589a3599fe08d7eb908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:00:55.631211 6638 raft_consensus.cc:729] T 286e5957de11442aa5b6773d7d5cb8f3 P 9535377387324589a3599fe08d7eb908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9535377387324589a3599fe08d7eb908, State: Initialized, Role: FOLLOWER
I20260806 01:00:55.631342 6638 consensus_queue.cc:260] T 286e5957de11442aa5b6773d7d5cb8f3 P 9535377387324589a3599fe08d7eb908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } }
I20260806 01:00:55.631690 6638 ts_tablet_manager.cc:1428] T 286e5957de11442aa5b6773d7d5cb8f3 P 9535377387324589a3599fe08d7eb908: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260806 01:00:55.631701 6484 heartbeater.cc:503] Master 127.23.242.62:43009 was elected leader, sending a full tablet report...
I20260806 01:00:55.633749 6642 raft_consensus.cc:348] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } }
I20260806 01:00:55.634054 6642 raft_consensus.cc:374] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:00:55.634099 6642 raft_consensus.cc:729] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45227c50fdc415c8cb398d7f36bfdeb, State: Initialized, Role: FOLLOWER
I20260806 01:00:55.634258 6642 consensus_queue.cc:260] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } }
I20260806 01:00:55.634752 6642 ts_tablet_manager.cc:1428] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260806 01:00:55.634932 6619 heartbeater.cc:503] Master 127.23.242.62:43009 was elected leader, sending a full tablet report...
W20260806 01:00:55.673432 6348 tablet.cc:2369] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 01:00:55.824352 6621 tablet.cc:2369] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 01:00:55.843286 6646 raft_consensus.cc:482] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 01:00:55.843406 6646 raft_consensus.cc:504] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } }
I20260806 01:00:55.843889 6646 leader_election.cc:288] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9535377387324589a3599fe08d7eb908 (127.23.242.2:37291), 9f248ea2790943f2b843ad446336501e (127.23.242.1:41361)
I20260806 01:00:55.848568 6647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:55.844050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:37291 (local address 127.23.242.3:47057)
0806 01:00:55.844442 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:55.844454 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:55.844520 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:55.844666 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:55.844667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:55.844668 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:00:55.844746 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:55.844747 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.845763 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.845764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:55.847192 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:55.847196 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.848224 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.848225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:55.848306 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:55.848311 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 01:00:55.848378 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 01:00:55.848423 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":182,"thread_start_us":135,"threads_started":1}
I20260806 01:00:55.848593 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:55.844141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:47057 (local address 127.23.242.2:37291)
0806 01:00:55.844201 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:55.844204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:55.844535 (+ 331us) server_negotiation.cc:407] Connection header received
0806 01:00:55.844575 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:55.844576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:55.844589 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:55.844652 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:00:55.844844 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.845656 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.847528 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.847987 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.848179 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:55.848192 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:55.848412 (+ 220us) server_negotiation.cc:299] Negotiation successful
0806 01:00:55.848479 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 01:00:55.849229 6438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286e5957de11442aa5b6773d7d5cb8f3" candidate_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9535377387324589a3599fe08d7eb908" is_pre_election: true
I20260806 01:00:55.849362 6622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:55.844045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:41361 (local address 127.23.242.3:46649)
0806 01:00:55.844182 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:55.844197 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:55.844250 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:55.844432 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:55.844433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:55.844435 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:00:55.844527 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:55.844530 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.845689 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.845690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:55.848042 (+ 2352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:55.848046 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.849124 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.849125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:55.849208 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:55.849212 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 01:00:55.849249 (+ 37us) client_negotiation.cc:240] Negotiation successful
0806 01:00:55.849296 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260806 01:00:55.849444 6438 raft_consensus.cc:2454] T 286e5957de11442aa5b6773d7d5cb8f3 P 9535377387324589a3599fe08d7eb908 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f45227c50fdc415c8cb398d7f36bfdeb in term 0.
I20260806 01:00:55.849707 6508 leader_election.cc:302] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9535377387324589a3599fe08d7eb908, f45227c50fdc415c8cb398d7f36bfdeb; no voters:
I20260806 01:00:55.849795 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:55.844227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:46649 (local address 127.23.242.1:41361)
0806 01:00:55.844275 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:55.844277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:55.844292 (+ 15us) server_negotiation.cc:407] Connection header received
0806 01:00:55.844326 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:55.844327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:55.844342 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:55.844413 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:00:55.844612 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.845586 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.848180 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.849002 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.849563 (+ 561us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:55.849578 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:55.849600 (+ 22us) server_negotiation.cc:299] Negotiation successful
0806 01:00:55.849662 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 01:00:55.849860 6646 raft_consensus.cc:2781] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 01:00:55.849908 6646 raft_consensus.cc:482] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 01:00:55.849934 6646 raft_consensus.cc:3037] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:00:55.850287 6301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286e5957de11442aa5b6773d7d5cb8f3" candidate_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f248ea2790943f2b843ad446336501e" is_pre_election: true
I20260806 01:00:55.850450 6301 raft_consensus.cc:2454] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f45227c50fdc415c8cb398d7f36bfdeb in term 0.
I20260806 01:00:55.851197 6646 raft_consensus.cc:504] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } }
I20260806 01:00:55.851369 6646 leader_election.cc:288] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb [CANDIDATE]: Term 1 election: Requested vote from peers 9535377387324589a3599fe08d7eb908 (127.23.242.2:37291), 9f248ea2790943f2b843ad446336501e (127.23.242.1:41361)
I20260806 01:00:55.851532 6438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286e5957de11442aa5b6773d7d5cb8f3" candidate_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9535377387324589a3599fe08d7eb908"
I20260806 01:00:55.851619 6301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "286e5957de11442aa5b6773d7d5cb8f3" candidate_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f248ea2790943f2b843ad446336501e"
I20260806 01:00:55.851652 6438 raft_consensus.cc:3037] T 286e5957de11442aa5b6773d7d5cb8f3 P 9535377387324589a3599fe08d7eb908 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:00:55.851727 6301 raft_consensus.cc:3037] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:00:55.852463 6438 raft_consensus.cc:2454] T 286e5957de11442aa5b6773d7d5cb8f3 P 9535377387324589a3599fe08d7eb908 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f45227c50fdc415c8cb398d7f36bfdeb in term 1.
I20260806 01:00:55.852661 6508 leader_election.cc:302] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9535377387324589a3599fe08d7eb908, f45227c50fdc415c8cb398d7f36bfdeb; no voters:
I20260806 01:00:55.852789 6646 raft_consensus.cc:2781] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:00:55.852941 6301 raft_consensus.cc:2454] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f45227c50fdc415c8cb398d7f36bfdeb in term 1.
I20260806 01:00:55.853008 6646 raft_consensus.cc:686] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb [term 1 LEADER]: Becoming Leader. State: Replica: f45227c50fdc415c8cb398d7f36bfdeb, State: Running, Role: LEADER
I20260806 01:00:55.853142 6646 consensus_queue.cc:237] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } }
I20260806 01:00:55.854342 6149 catalog_manager.cc:5495] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb reported cstate change: term changed from 0 to 1, leader changed from <none> to f45227c50fdc415c8cb398d7f36bfdeb (127.23.242.3). New cstate: current_term: 1 leader_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } health_report { overall_health: UNKNOWN } } }
W20260806 01:00:55.881112 6485 tablet.cc:2369] T 286e5957de11442aa5b6773d7d5cb8f3 P 9535377387324589a3599fe08d7eb908: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 01:00:55.905669 6635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:55.902491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49910 (local address 127.23.242.3:34067)
0806 01:00:55.902568 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:55.902570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:55.902607 (+ 37us) server_negotiation.cc:407] Connection header received
0806 01:00:55.902772 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:55.902773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:55.902795 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:55.902884 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 01:00:55.903152 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.903687 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.904772 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:55.904997 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:55.905085 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:55.905292 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 01:00:55.905406 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 01:00:55.905511 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:55.905534 (+ 23us) server_negotiation.cc:299] Negotiation successful
0806 01:00:55.905559 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 01:00:55.909312 6301 raft_consensus.cc:1261] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e [term 1 FOLLOWER]: Refusing update from remote peer f45227c50fdc415c8cb398d7f36bfdeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:00:55.909991 6438 raft_consensus.cc:1261] T 286e5957de11442aa5b6773d7d5cb8f3 P 9535377387324589a3599fe08d7eb908 [term 1 FOLLOWER]: Refusing update from remote peer f45227c50fdc415c8cb398d7f36bfdeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:00:55.910132 6649 consensus_queue.cc:1035] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:00:55.910362 6649 consensus_queue.cc:1035] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:00:55.913256 6652 mvcc.cc:205] Tried to move back new op lower bound from 7315366116998639616 to 7315366116775149568. Current Snapshot: MvccSnapshot[applied={T|T < 7315366116998639616}]
I20260806 01:00:55.913695 6654 mvcc.cc:205] Tried to move back new op lower bound from 7315366116998639616 to 7315366116775149568. Current Snapshot: MvccSnapshot[applied={T|T < 7315366116998639616}]
I20260806 01:00:55.915586 6653 mvcc.cc:205] Tried to move back new op lower bound from 7315366116998639616 to 7315366116775149568. Current Snapshot: MvccSnapshot[applied={T|T < 7315366116998639616 or (T in {7315366116998639616})}]
I20260806 01:00:55.917042 6150 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 01:00:55.917155 6150 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 01:00:55.918481 6150 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875: Sending DeleteTablet for 3 replicas of tablet 286e5957de11442aa5b6773d7d5cb8f3
I20260806 01:00:55.919176 6279 tablet_service.cc:1514] Processing DeleteTablet for tablet 286e5957de11442aa5b6773d7d5cb8f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:00:55 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:41118
I20260806 01:00:55.919211 6416 tablet_service.cc:1514] Processing DeleteTablet for tablet 286e5957de11442aa5b6773d7d5cb8f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:00:55 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:33164
I20260806 01:00:55.919202 6553 tablet_service.cc:1514] Processing DeleteTablet for tablet 286e5957de11442aa5b6773d7d5cb8f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:00:55 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:49904
I20260806 01:00:55.919314 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 6214
I20260806 01:00:55.922159 6664 tablet_replica.cc:330] T 286e5957de11442aa5b6773d7d5cb8f3 P 9535377387324589a3599fe08d7eb908: stopping tablet replica
I20260806 01:00:55.922374 6664 raft_consensus.cc:2229] T 286e5957de11442aa5b6773d7d5cb8f3 P 9535377387324589a3599fe08d7eb908 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 01:00:55.922567 6664 raft_consensus.cc:2258] T 286e5957de11442aa5b6773d7d5cb8f3 P 9535377387324589a3599fe08d7eb908 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 01:00:55.924505 6664 ts_tablet_manager.cc:1905] T 286e5957de11442aa5b6773d7d5cb8f3 P 9535377387324589a3599fe08d7eb908: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 01:00:55.926800 6662 tablet_replica.cc:330] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb: stopping tablet replica
I20260806 01:00:55.927040 6662 raft_consensus.cc:2229] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb [term 1 LEADER]: Raft consensus shutting down.
I20260806 01:00:55.927278 6662 raft_consensus.cc:2258] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 01:00:55.928012 6662 ts_tablet_manager.cc:1905] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 01:00:55.929409 6664 ts_tablet_manager.cc:1918] T 286e5957de11442aa5b6773d7d5cb8f3 P 9535377387324589a3599fe08d7eb908: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 01:00:55.929519 6664 log.cc:1196] T 286e5957de11442aa5b6773d7d5cb8f3 P 9535377387324589a3599fe08d7eb908: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/wal/wals/286e5957de11442aa5b6773d7d5cb8f3
I20260806 01:00:55.929970 6664 ts_tablet_manager.cc:1939] T 286e5957de11442aa5b6773d7d5cb8f3 P 9535377387324589a3599fe08d7eb908: Deleting consensus metadata
I20260806 01:00:55.930346 6662 ts_tablet_manager.cc:1918] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 01:00:55.930506 6662 log.cc:1196] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/wal/wals/286e5957de11442aa5b6773d7d5cb8f3
I20260806 01:00:55.930752 6137 catalog_manager.cc:4841] TS 9535377387324589a3599fe08d7eb908 (127.23.242.2:37291): tablet 286e5957de11442aa5b6773d7d5cb8f3 (table test-table [id=3d6c8bf17cb246daaaad48208ceb7a77]) successfully deleted
I20260806 01:00:55.931171 6662 ts_tablet_manager.cc:1939] T 286e5957de11442aa5b6773d7d5cb8f3 P f45227c50fdc415c8cb398d7f36bfdeb: Deleting consensus metadata
W20260806 01:00:55.931890 6135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.242.1:41361 (error 108)
I20260806 01:00:55.932037 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 6352
I20260806 01:00:55.936720 6636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:55.936486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:41361 (local address 127.0.0.1:41132)
0806 01:00:55.936589 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:55.936670 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.1:41361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260806 01:00:55.937057 6135 catalog_manager.cc:4584] TS 9f248ea2790943f2b843ad446336501e (127.23.242.1:41361): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 286e5957de11442aa5b6773d7d5cb8f3: Network error: Client connection negotiation failed: client connection to 127.23.242.1:41361: connect: Connection refused (error 111)
I20260806 01:00:55.939267 6135 catalog_manager.cc:4841] TS f45227c50fdc415c8cb398d7f36bfdeb (127.23.242.3:34067): tablet 286e5957de11442aa5b6773d7d5cb8f3 (table test-table [id=3d6c8bf17cb246daaaad48208ceb7a77]) successfully deleted
I20260806 01:00:55.940819 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 6489
I20260806 01:00:55.948691 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 6117
I20260806 01:00:55.957321 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:43009
--webserver_interface=127.23.242.62
--webserver_port=45829
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:41023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:43009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:00:56.060761 6666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:00:56.061014 6666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:00:56.061043 6666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:00:56.062306 6666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:00:56.062351 6666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:00:56.062366 6666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:00:56.062381 6666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:00:56.062392 6666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:00:56.064026 6666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:43009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:43009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=45829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:00:56.064533 6666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:00:56.065068 6666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:00:56.076181 6672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:56.076316 6673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:56.076149 6675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:56.076514 6666 server_base.cc:1034] running on GCE node
I20260806 01:00:56.076835 6666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:00:56.077204 6666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:00:56.078375 6666 hybrid_clock.cc:648] HybridClock initialized: now 1785978056078351 us; error 39 us; skew 500 ppm
Aug 06 01:00:56 dist-test-slave-dhph krb5kdc[6104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978056, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:00:56.081387 6666 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 01:00:56.081835 6666 webserver.cc:460] Webserver started at http://127.23.242.62:45829/ using document root <none> and password file <none>
I20260806 01:00:56.082116 6666 fs_manager.cc:359] Metadata directory not provided
I20260806 01:00:56.082180 6666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:00:56.083950 6666 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:00:56.084972 6682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:56.085377 6666 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:00:56.085466 6666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "ef9c497be1c44b0e80cdbbb063aac875"
format_stamp: "Formatted at 2026-08-06 01:00:54 on dist-test-slave-dhph"
server_key: "ada6fa0f6ac4f6c732770b82dc5d2a25"
server_key_iv: "137db35b00a0673397254544ae372e64"
server_key_version: "encryptionkey@0"
I20260806 01:00:56.086037 6666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:00:56.105800 6666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:00:56.109380 6666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:00:56.109988 6666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:00:56.116078 6666 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:43009
I20260806 01:00:56.116212 6736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:43009 every 8 connection(s)
I20260806 01:00:56.116597 6666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:00:56.118376 6737 sys_catalog.cc:263] Verifying existing consensus state
I20260806 01:00:56.119200 6737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875: Bootstrap starting.
I20260806 01:00:56.126349 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 6666
I20260806 01:00:56.126758 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:41361
--local_ip_for_outbound_sockets=127.23.242.1
--tserver_master_addrs=127.23.242.62:43009
--webserver_port=35431
--webserver_interface=127.23.242.1
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:41023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 01:00:56.129688 6737 log.cc:824] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875: Log is configured to *not* fsync() on all Append() calls
I20260806 01:00:56.132418 6737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 01:00:56.132736 6737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875: Bootstrap complete.
I20260806 01:00:56.135313 6737 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9c497be1c44b0e80cdbbb063aac875" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43009 } }
I20260806 01:00:56.135568 6737 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef9c497be1c44b0e80cdbbb063aac875, State: Initialized, Role: FOLLOWER
I20260806 01:00:56.135787 6737 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9c497be1c44b0e80cdbbb063aac875" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43009 } }
I20260806 01:00:56.135886 6737 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:00:56.135948 6737 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:00:56.135998 6737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [term 1 FOLLOWER]: Advancing to term 2
I20260806 01:00:56.137039 6737 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9c497be1c44b0e80cdbbb063aac875" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43009 } }
I20260806 01:00:56.137166 6737 leader_election.cc:302] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef9c497be1c44b0e80cdbbb063aac875; no voters:
I20260806 01:00:56.137362 6737 leader_election.cc:288] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 01:00:56.137459 6741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 01:00:56.137655 6737 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:00:56.137670 6741 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [term 2 LEADER]: Becoming Leader. State: Replica: ef9c497be1c44b0e80cdbbb063aac875, State: Running, Role: LEADER
I20260806 01:00:56.137802 6741 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9c497be1c44b0e80cdbbb063aac875" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43009 } }
I20260806 01:00:56.138238 6743 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef9c497be1c44b0e80cdbbb063aac875. Latest consensus state: current_term: 2 leader_uuid: "ef9c497be1c44b0e80cdbbb063aac875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9c497be1c44b0e80cdbbb063aac875" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43009 } } }
I20260806 01:00:56.138391 6743 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [sys.catalog]: This master's current role is: LEADER
I20260806 01:00:56.138216 6742 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ef9c497be1c44b0e80cdbbb063aac875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9c497be1c44b0e80cdbbb063aac875" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43009 } } }
I20260806 01:00:56.138700 6742 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [sys.catalog]: This master's current role is: LEADER
I20260806 01:00:56.139228 6750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:00:56.140148 6750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:00:56.140398 6750 catalog_manager.cc:1259] Loaded cluster ID: 32b0a8237d7e4979ae20796c586c2c7f
I20260806 01:00:56.140465 6750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:00:56.140722 6750 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 01:00:56.144479 6750 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 01:00:56.147909 6750 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:00:56.148216 6750 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875: Loaded TSK: 0
I20260806 01:00:56.148468 6750 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 01:00:56.251523 6739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:00:56.252138 6739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:00:56.252192 6739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:00:56.254310 6739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:00:56.254415 6739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:00:56.254475 6739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 01:00:56.256611 6739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:41361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=35431
--tserver_master_addrs=127.23.242.62:43009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:00:56.257014 6739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:00:56.257313 6739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:00:56.264483 6767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:56.264550 6765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:56.264623 6764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:56.265112 6739 server_base.cc:1034] running on GCE node
I20260806 01:00:56.265367 6739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:00:56.265657 6739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:00:56.266896 6739 hybrid_clock.cc:648] HybridClock initialized: now 1785978056266873 us; error 43 us; skew 500 ppm
Aug 06 01:00:56 dist-test-slave-dhph krb5kdc[6104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978056, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:00:56.269878 6739 init.cc:377] Logged in from keytab as kudu/127.23.242.1@KRBTEST.COM (short username kudu)
I20260806 01:00:56.270494 6739 webserver.cc:460] Webserver started at http://127.23.242.1:35431/ using document root <none> and password file <none>
I20260806 01:00:56.270860 6739 fs_manager.cc:359] Metadata directory not provided
I20260806 01:00:56.270962 6739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:00:56.272513 6739 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 01:00:56.273448 6774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:56.273699 6739 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 01:00:56.273856 6739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "9f248ea2790943f2b843ad446336501e"
format_stamp: "Formatted at 2026-08-06 01:00:55 on dist-test-slave-dhph"
server_key: "2a2d20195556cab78e89075edfc29feb"
server_key_iv: "3d364d3cd187e0345f3d48a77cc70c9b"
server_key_version: "encryptionkey@0"
I20260806 01:00:56.274214 6739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:00:56.298027 6739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:00:56.301234 6739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:00:56.301438 6739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:00:56.301755 6739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:00:56.302300 6781 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 01:00:56.304093 6739 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 01:00:56.304159 6739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260806 01:00:56.304189 6739 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 01:00:56.304924 6739 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 01:00:56.304972 6739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260806 01:00:56.305024 6781 tablet_bootstrap.cc:492] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e: Bootstrap starting.
I20260806 01:00:56.316864 6739 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:41361
I20260806 01:00:56.317467 6739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 01:00:56.318673 6890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:41361 every 8 connection(s)
I20260806 01:00:56.321206 6781 log.cc:824] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e: Log is configured to *not* fsync() on all Append() calls
I20260806 01:00:56.323699 6781 tablet_bootstrap.cc:492] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 01:00:56.324028 6781 tablet_bootstrap.cc:492] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e: Bootstrap complete.
I20260806 01:00:56.325146 6781 ts_tablet_manager.cc:1397] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.013s
I20260806 01:00:56.326341 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 6739
I20260806 01:00:56.326706 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:37291
--local_ip_for_outbound_sockets=127.23.242.2
--tserver_master_addrs=127.23.242.62:43009
--webserver_port=36743
--webserver_interface=127.23.242.2
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:41023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 01:00:56.326710 6781 raft_consensus.cc:348] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } }
I20260806 01:00:56.327149 6781 raft_consensus.cc:729] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f248ea2790943f2b843ad446336501e, State: Initialized, Role: FOLLOWER
I20260806 01:00:56.327309 6781 consensus_queue.cc:260] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } }
I20260806 01:00:56.327664 6781 ts_tablet_manager.cc:1428] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
Aug 06 01:00:56 dist-test-slave-dhph krb5kdc[6104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978056, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:00:56.334043 6894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:56.318387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:60917 (local address 127.23.242.62:43009)
0806 01:00:56.318918 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:56.318926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:56.319187 (+ 261us) server_negotiation.cc:407] Connection header received
0806 01:00:56.320371 (+ 1184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:56.320377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:56.320430 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:56.320571 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:56.321597 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.322997 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.324446 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.325121 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.325284 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:56.328653 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:56.328661 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:56.328661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:56.328682 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:56.331318 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:56.331916 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:56.331921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:56.331921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:56.332058 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:56.332355 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:56.332356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:56.332357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:56.332941 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:56.333141 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:56.333567 (+ 426us) server_negotiation.cc:299] Negotiation successful
0806 01:00:56.333773 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":91,"threads_started":1}
I20260806 01:00:56.334044 6893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:56.318264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.1:60917)
0806 01:00:56.318878 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:56.318947 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:56.319911 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:56.320651 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:56.320656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:56.321105 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:00:56.321408 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:56.321417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.323139 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.323141 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:56.324336 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:56.324340 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.325298 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.325300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:56.325454 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:56.326247 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:00:56.326258 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:00:56.328419 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:00:56.331434 (+ 3015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:56.331443 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:56.331445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:56.331786 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:56.332156 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:56.332158 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:56.332158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:56.332272 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:56.333124 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:00:56.333140 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:00:56.333342 (+ 202us) client_negotiation.cc:769] Sending connection context
0806 01:00:56.333486 (+ 144us) client_negotiation.cc:240] Negotiation successful
0806 01:00:56.333770 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":105,"threads_started":1}
I20260806 01:00:56.335201 6891 heartbeater.cc:347] Connected to a master server at 127.23.242.62:43009
I20260806 01:00:56.335320 6891 heartbeater.cc:464] Registering TS with master...
I20260806 01:00:56.335554 6891 heartbeater.cc:511] Master 127.23.242.62:43009 requested a full tablet report, sending...
I20260806 01:00:56.336513 6699 ts_manager.cc:194] Registered new tserver with Master: 9f248ea2790943f2b843ad446336501e (127.23.242.1:41361)
I20260806 01:00:56.338017 6699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.1@KRBTEST.COM'} at 127.23.242.1:60917
I20260806 01:00:56.339046 6891 heartbeater.cc:503] Master 127.23.242.62:43009 was elected leader, sending a full tablet report...
Aug 06 01:00:56 dist-test-slave-dhph krb5kdc[6104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978056, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for kudu/127.23.242.1@KRBTEST.COM
I20260806 01:00:56.348390 6898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:56.337318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:41361 (local address 127.0.0.1:41136)
0806 01:00:56.337577 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:56.337611 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:56.337690 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:56.338083 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:56.338088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:56.338499 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:00:56.339299 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:56.339302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.340295 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.340296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:56.341469 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:56.341473 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.342268 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.342269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:56.342368 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:56.342986 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:00:56.342992 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:00:56.345119 (+ 2127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:00:56.347122 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:56.347131 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:56.347133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:56.347343 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:56.347581 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:56.347582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:56.347582 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:56.347648 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:56.347955 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:00:56.347962 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:00:56.348022 (+ 60us) client_negotiation.cc:769] Sending connection context
0806 01:00:56.348139 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 01:00:56.348283 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":120,"threads_started":1}
I20260806 01:00:56.348390 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:56.337414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41136 (local address 127.23.242.1:41361)
0806 01:00:56.337625 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:56.337630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:56.337667 (+ 37us) server_negotiation.cc:407] Connection header received
0806 01:00:56.337924 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:56.337935 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:56.337962 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:56.338071 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:56.339429 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.340196 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.341593 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.342109 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.342237 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:56.345233 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:56.345243 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:56.345244 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:56.345253 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:56.347004 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:56.347433 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:56.347437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:56.347437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:56.347501 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:56.347710 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:56.347711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:56.347711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:56.347872 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:56.347966 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:56.348181 (+ 215us) server_negotiation.cc:299] Negotiation successful
0806 01:00:56.348282 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":97,"threads_started":1}
I20260806 01:00:56.348954 6824 tablet_service.cc:1514] Processing DeleteTablet for tablet 286e5957de11442aa5b6773d7d5cb8f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:00:55 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:41136
I20260806 01:00:56.349675 6900 tablet_replica.cc:330] stopping tablet replica
I20260806 01:00:56.349821 6900 raft_consensus.cc:2229] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 01:00:56.349891 6900 raft_consensus.cc:2258] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 01:00:56.350286 6900 ts_tablet_manager.cc:1905] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 01:00:56.353323 6900 ts_tablet_manager.cc:1918] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 01:00:56.353421 6900 log.cc:1196] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/wal/wals/286e5957de11442aa5b6773d7d5cb8f3
I20260806 01:00:56.353681 6900 ts_tablet_manager.cc:1939] T 286e5957de11442aa5b6773d7d5cb8f3 P 9f248ea2790943f2b843ad446336501e: Deleting consensus metadata
I20260806 01:00:56.354323 6684 catalog_manager.cc:4841] TS 9f248ea2790943f2b843ad446336501e (127.23.242.1:41361): tablet 286e5957de11442aa5b6773d7d5cb8f3 (table test-table [id=3d6c8bf17cb246daaaad48208ceb7a77]) successfully deleted
W20260806 01:00:56.431788 6897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:00:56.432013 6897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:00:56.432039 6897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:00:56.433514 6897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:00:56.433593 6897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:00:56.433648 6897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 01:00:56.435856 6897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:37291
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=36743
--tserver_master_addrs=127.23.242.62:43009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:00:56.436086 6897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:00:56.436292 6897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:00:56.442930 6905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:56.442958 6906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:56.443027 6908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:56.443812 6897 server_base.cc:1034] running on GCE node
I20260806 01:00:56.444092 6897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:00:56.444353 6897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:00:56.445582 6897 hybrid_clock.cc:648] HybridClock initialized: now 1785978056445519 us; error 90 us; skew 500 ppm
Aug 06 01:00:56 dist-test-slave-dhph krb5kdc[6104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978056, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:00:56.449360 6897 init.cc:377] Logged in from keytab as kudu/127.23.242.2@KRBTEST.COM (short username kudu)
I20260806 01:00:56.450367 6897 webserver.cc:460] Webserver started at http://127.23.242.2:36743/ using document root <none> and password file <none>
I20260806 01:00:56.450748 6897 fs_manager.cc:359] Metadata directory not provided
I20260806 01:00:56.450853 6897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:00:56.452931 6897 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 01:00:56.454195 6915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:56.454838 6897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 01:00:56.454974 6897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "9535377387324589a3599fe08d7eb908"
format_stamp: "Formatted at 2026-08-06 01:00:55 on dist-test-slave-dhph"
server_key: "2162ed441f203abeccede0ce1e71fcf2"
server_key_iv: "87a15761a8ec33827ac3242d36405cf0"
server_key_version: "encryptionkey@0"
I20260806 01:00:56.455428 6897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:00:56.481905 6897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:00:56.484400 6897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:00:56.484622 6897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:00:56.484890 6897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:00:56.485210 6897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:00:56.485265 6897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:56.485311 6897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:00:56.485334 6897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:56.492494 6897 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:37291
I20260806 01:00:56.492779 7033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:37291 every 8 connection(s)
I20260806 01:00:56.493057 6897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 01:00:56.496269 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 6897
I20260806 01:00:56.496634 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:34067
--local_ip_for_outbound_sockets=127.23.242.3
--tserver_master_addrs=127.23.242.62:43009
--webserver_port=46735
--webserver_interface=127.23.242.3
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:41023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 01:00:56 dist-test-slave-dhph krb5kdc[6104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978056, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:00:56.507249 6894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:56.494044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:43785 (local address 127.23.242.62:43009)
0806 01:00:56.494100 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:56.494102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:56.494348 (+ 246us) server_negotiation.cc:407] Connection header received
0806 01:00:56.495084 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:56.495085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:56.495098 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:56.495191 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:56.496311 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.497144 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.498673 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.498908 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.498974 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:56.502829 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:56.502831 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:56.502831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:56.502840 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:56.504547 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:56.505393 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:56.505394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:56.505394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:56.505453 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:56.506157 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:56.506158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:56.506159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:56.506311 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:56.506382 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:56.507072 (+ 690us) server_negotiation.cc:299] Negotiation successful
0806 01:00:56.507174 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 01:00:56.507592 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:56.493965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.2:43785)
0806 01:00:56.494181 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:56.494230 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:56.494954 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:56.495302 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:56.495306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:56.495909 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:00:56.496182 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:56.496187 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.497280 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.497282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:56.498502 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:56.498506 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.499005 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.499006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:56.499174 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:56.500192 (+ 1018us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:00:56.500210 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:00:56.502675 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:00:56.504672 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:56.504681 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:56.504684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:56.505094 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:56.505541 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:56.505542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:56.505542 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:56.505946 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:56.506416 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:00:56.506426 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:00:56.506842 (+ 416us) client_negotiation.cc:769] Sending connection context
0806 01:00:56.507003 (+ 161us) client_negotiation.cc:240] Negotiation successful
0806 01:00:56.507250 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":81,"threads_started":1}
I20260806 01:00:56.508456 7034 heartbeater.cc:347] Connected to a master server at 127.23.242.62:43009
I20260806 01:00:56.508895 7034 heartbeater.cc:464] Registering TS with master...
I20260806 01:00:56.509270 7034 heartbeater.cc:511] Master 127.23.242.62:43009 requested a full tablet report, sending...
I20260806 01:00:56.509981 6699 ts_manager.cc:194] Registered new tserver with Master: 9535377387324589a3599fe08d7eb908 (127.23.242.2:37291)
I20260806 01:00:56.510859 6699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.2@KRBTEST.COM'} at 127.23.242.2:43785
W20260806 01:00:56.607884 7037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:00:56.608125 7037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:00:56.608155 7037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:00:56.609407 7037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:00:56.609455 7037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:00:56.609485 7037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 01:00:56.613545 7037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:34067
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=46735
--tserver_master_addrs=127.23.242.62:43009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:00:56.614941 7037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:00:56.615275 7037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:00:56.622135 7042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:56.622222 7045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:56.622274 7043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:56.622367 7037 server_base.cc:1034] running on GCE node
I20260806 01:00:56.622663 7037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:00:56.622949 7037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:00:56.624150 7037 hybrid_clock.cc:648] HybridClock initialized: now 1785978056624125 us; error 42 us; skew 500 ppm
Aug 06 01:00:56 dist-test-slave-dhph krb5kdc[6104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978056, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:00:56.627337 7037 init.cc:377] Logged in from keytab as kudu/127.23.242.3@KRBTEST.COM (short username kudu)
I20260806 01:00:56.627871 7037 webserver.cc:460] Webserver started at http://127.23.242.3:46735/ using document root <none> and password file <none>
I20260806 01:00:56.628137 7037 fs_manager.cc:359] Metadata directory not provided
I20260806 01:00:56.628230 7037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:00:56.629793 7037 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260806 01:00:56.630879 7052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:56.631086 7037 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:00:56.631196 7037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "f45227c50fdc415c8cb398d7f36bfdeb"
format_stamp: "Formatted at 2026-08-06 01:00:55 on dist-test-slave-dhph"
server_key: "2c9d7693ce1ef7f0243182d01de82540"
server_key_iv: "8a151c8ca02db6200627325899c2b267"
server_key_version: "encryptionkey@0"
I20260806 01:00:56.631575 7037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:00:56.646224 7037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:00:56.649703 7037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:00:56.649852 7037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:00:56.650182 7037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:00:56.650731 7037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:00:56.650779 7037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:56.650805 7037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:00:56.650820 7037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:56.658979 7037 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:34067
I20260806 01:00:56.659047 7168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:34067 every 8 connection(s)
I20260806 01:00:56.659437 7037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
I20260806 01:00:56.665781 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 7037
Aug 06 01:00:56 dist-test-slave-dhph krb5kdc[6104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978056, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:00:56.671674 6894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:56.660368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:59773 (local address 127.23.242.62:43009)
0806 01:00:56.660462 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:56.660464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:56.660929 (+ 465us) server_negotiation.cc:407] Connection header received
0806 01:00:56.661736 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:56.661737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:56.661750 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:56.661808 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:56.663205 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.663728 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.665165 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.665426 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.665499 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:56.668080 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:56.668081 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:56.668082 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:56.668090 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:56.669520 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:56.670267 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:56.670268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:56.670269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:56.670336 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:56.670736 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:56.670738 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:56.670738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:56.670860 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:56.670962 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:56.671345 (+ 383us) server_negotiation.cc:299] Negotiation successful
0806 01:00:56.671491 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 01:00:56.671766 7171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:56.660341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.3:59773)
0806 01:00:56.660722 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:56.660763 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:56.661556 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:56.661896 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:56.661900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:56.662765 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:00:56.662992 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:56.662998 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.663832 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.663833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:56.665019 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:56.665022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.665559 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.665561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:56.665734 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:56.666446 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:00:56.666454 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:00:56.667976 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:00:56.669656 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:56.669664 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:56.669666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:56.670159 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:56.670428 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:56.670429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:56.670430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:56.670658 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:56.670954 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:00:56.670962 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:00:56.671146 (+ 184us) client_negotiation.cc:769] Sending connection context
0806 01:00:56.671289 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 01:00:56.671501 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":218,"threads_started":1}
I20260806 01:00:56.672381 7169 heartbeater.cc:347] Connected to a master server at 127.23.242.62:43009
I20260806 01:00:56.672518 7169 heartbeater.cc:464] Registering TS with master...
I20260806 01:00:56.672739 7169 heartbeater.cc:511] Master 127.23.242.62:43009 requested a full tablet report, sending...
I20260806 01:00:56.673280 6699 ts_manager.cc:194] Registered new tserver with Master: f45227c50fdc415c8cb398d7f36bfdeb (127.23.242.3:34067)
I20260806 01:00:56.673892 6699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.3@KRBTEST.COM'} at 127.23.242.3:59773
I20260806 01:00:56.674464 7172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:56.666251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39046 (local address 127.23.242.62:43009)
0806 01:00:56.666393 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:56.666395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:56.666526 (+ 131us) server_negotiation.cc:407] Connection header received
0806 01:00:56.666663 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:56.666664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:56.666677 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:56.666757 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:56.667567 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.668187 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.669014 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.669251 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.669337 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:56.670927 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:56.670929 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:56.670929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:56.670974 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:56.672981 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:56.673467 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:56.673468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:56.673468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:56.673532 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:56.673752 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:56.673753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:56.673753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:56.673901 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:56.674103 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:56.674231 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 01:00:56.674337 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260806 01:00:56.675644 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 01:00:56.679315 7172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:56.676056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39060 (local address 127.23.242.62:43009)
0806 01:00:56.676134 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:56.676136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:56.676248 (+ 112us) server_negotiation.cc:407] Connection header received
0806 01:00:56.676349 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:56.676350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:56.676388 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:56.676481 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 01:00:56.676618 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.677163 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.678282 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.678529 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.678671 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:56.678863 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 01:00:56.678976 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 01:00:56.679071 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:56.679117 (+ 46us) server_negotiation.cc:299] Negotiation successful
0806 01:00:56.679144 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 01:00:56.679934 6699 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 01:00:56.681742 6699 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 01:00:56.685770 6824 tablet_service.cc:1467] Processing CreateTablet for tablet 977b8f0507154730b9186f6113edfb70 (DEFAULT_TABLE table=test-table [id=bb3da9ba9b454a7b93baf28f890c997a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:00:56.686002 6824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 977b8f0507154730b9186f6113edfb70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:00:56.687616 6781 tablet_bootstrap.cc:492] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e: Bootstrap starting.
I20260806 01:00:56.687954 6781 tablet_bootstrap.cc:654] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e: Neither blocks nor log segments found. Creating new log.
I20260806 01:00:56.688601 6781 tablet_bootstrap.cc:492] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e: No bootstrap required, opened a new log
I20260806 01:00:56.688676 6781 ts_tablet_manager.cc:1397] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 01:00:56.689038 6781 raft_consensus.cc:348] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } }
I20260806 01:00:56.689124 6781 raft_consensus.cc:374] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:00:56.689194 6781 raft_consensus.cc:729] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f248ea2790943f2b843ad446336501e, State: Initialized, Role: FOLLOWER
I20260806 01:00:56.689240 6781 consensus_queue.cc:260] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } }
I20260806 01:00:56.689339 6781 ts_tablet_manager.cc:1428] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 01:00:56.690286 6898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:56.685318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:34067 (local address 127.0.0.1:49918)
0806 01:00:56.685486 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:56.685500 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:56.685546 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:56.685944 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:56.685945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:56.685948 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:00:56.686016 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:56.686019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.687554 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.687556 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:56.688893 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:56.688895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.689981 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.689982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:56.690069 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:56.690092 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 01:00:56.690154 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 01:00:56.690204 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260806 01:00:56.690297 7180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:56.685445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49918 (local address 127.23.242.3:34067)
0806 01:00:56.685751 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:56.685755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:56.685765 (+ 10us) server_negotiation.cc:407] Connection header received
0806 01:00:56.685804 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:56.685806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:56.685822 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:56.685937 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:00:56.686128 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.687392 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.689012 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.689864 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.689956 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:56.690023 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:56.690189 (+ 166us) server_negotiation.cc:299] Negotiation successful
0806 01:00:56.690226 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":193,"threads_started":1}
I20260806 01:00:56.691128 7102 tablet_service.cc:1467] Processing CreateTablet for tablet 977b8f0507154730b9186f6113edfb70 (DEFAULT_TABLE table=test-table [id=bb3da9ba9b454a7b93baf28f890c997a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:00:56.691462 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:56.685323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:37291 (local address 127.0.0.1:33166)
0806 01:00:56.686683 (+ 1360us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:56.686694 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:56.686760 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:56.686975 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:56.686976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:56.686979 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:00:56.687025 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:56.687027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.688682 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.688683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:56.689957 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:56.689961 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.690995 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.690996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:56.691181 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:56.691187 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 01:00:56.691272 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 01:00:56.691373 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1334,"mutex_wait_us":13,"thread_start_us":102,"threads_started":1}
I20260806 01:00:56.691568 7102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 977b8f0507154730b9186f6113edfb70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:00:56.691501 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:56.685525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33166 (local address 127.23.242.2:37291)
0806 01:00:56.685713 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:56.685718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:56.686776 (+ 1058us) server_negotiation.cc:407] Connection header received
0806 01:00:56.686842 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:56.686845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:56.686863 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:56.686968 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:00:56.687104 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.688539 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.690049 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.690900 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.690960 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:56.690994 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:56.691318 (+ 324us) server_negotiation.cc:299] Negotiation successful
0806 01:00:56.691399 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":83,"threads_started":1}
I20260806 01:00:56.692314 6966 tablet_service.cc:1467] Processing CreateTablet for tablet 977b8f0507154730b9186f6113edfb70 (DEFAULT_TABLE table=test-table [id=bb3da9ba9b454a7b93baf28f890c997a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:00:56.693404 6966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 977b8f0507154730b9186f6113edfb70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:00:56.696808 7182 tablet_bootstrap.cc:492] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb: Bootstrap starting.
I20260806 01:00:56.697516 7182 tablet_bootstrap.cc:654] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb: Neither blocks nor log segments found. Creating new log.
I20260806 01:00:56.698289 7182 log.cc:824] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb: Log is configured to *not* fsync() on all Append() calls
I20260806 01:00:56.698518 7183 tablet_bootstrap.cc:492] T 977b8f0507154730b9186f6113edfb70 P 9535377387324589a3599fe08d7eb908: Bootstrap starting.
I20260806 01:00:56.699157 7182 tablet_bootstrap.cc:492] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb: No bootstrap required, opened a new log
I20260806 01:00:56.699193 7183 tablet_bootstrap.cc:654] T 977b8f0507154730b9186f6113edfb70 P 9535377387324589a3599fe08d7eb908: Neither blocks nor log segments found. Creating new log.
I20260806 01:00:56.699265 7182 ts_tablet_manager.cc:1397] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 01:00:56.699712 7183 log.cc:824] T 977b8f0507154730b9186f6113edfb70 P 9535377387324589a3599fe08d7eb908: Log is configured to *not* fsync() on all Append() calls
I20260806 01:00:56.700465 7183 tablet_bootstrap.cc:492] T 977b8f0507154730b9186f6113edfb70 P 9535377387324589a3599fe08d7eb908: No bootstrap required, opened a new log
I20260806 01:00:56.700523 7183 ts_tablet_manager.cc:1397] T 977b8f0507154730b9186f6113edfb70 P 9535377387324589a3599fe08d7eb908: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 01:00:56.700961 7182 raft_consensus.cc:348] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } }
I20260806 01:00:56.701131 7182 raft_consensus.cc:374] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:00:56.701213 7182 raft_consensus.cc:729] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45227c50fdc415c8cb398d7f36bfdeb, State: Initialized, Role: FOLLOWER
I20260806 01:00:56.701529 7182 consensus_queue.cc:260] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } }
I20260806 01:00:56.701897 7182 ts_tablet_manager.cc:1428] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 01:00:56.702152 7169 heartbeater.cc:503] Master 127.23.242.62:43009 was elected leader, sending a full tablet report...
I20260806 01:00:56.702931 7183 raft_consensus.cc:348] T 977b8f0507154730b9186f6113edfb70 P 9535377387324589a3599fe08d7eb908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } }
I20260806 01:00:56.703140 7183 raft_consensus.cc:374] T 977b8f0507154730b9186f6113edfb70 P 9535377387324589a3599fe08d7eb908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:00:56.703187 7183 raft_consensus.cc:729] T 977b8f0507154730b9186f6113edfb70 P 9535377387324589a3599fe08d7eb908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9535377387324589a3599fe08d7eb908, State: Initialized, Role: FOLLOWER
I20260806 01:00:56.703325 7183 consensus_queue.cc:260] T 977b8f0507154730b9186f6113edfb70 P 9535377387324589a3599fe08d7eb908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } }
I20260806 01:00:56.703665 7183 ts_tablet_manager.cc:1428] T 977b8f0507154730b9186f6113edfb70 P 9535377387324589a3599fe08d7eb908: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 01:00:56.703747 7034 heartbeater.cc:503] Master 127.23.242.62:43009 was elected leader, sending a full tablet report...
W20260806 01:00:56.743839 7035 tablet.cc:2369] T 977b8f0507154730b9186f6113edfb70 P 9535377387324589a3599fe08d7eb908: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 01:00:56.819368 6892 tablet.cc:2369] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 01:00:56.889619 7186 raft_consensus.cc:482] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 01:00:56.889937 7186 raft_consensus.cc:504] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } }
I20260806 01:00:56.890440 7186 leader_election.cc:288] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9535377387324589a3599fe08d7eb908 (127.23.242.2:37291), 9f248ea2790943f2b843ad446336501e (127.23.242.1:41361)
I20260806 01:00:56.894323 7171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:56.890756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:41361 (local address 127.23.242.3:43851)
0806 01:00:56.890825 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:56.890839 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:56.890892 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:56.891438 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:56.891439 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:56.891441 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:00:56.891493 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:56.891494 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.892435 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.892436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:56.893305 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:56.893307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.894029 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.894030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:56.894132 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:56.894137 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 01:00:56.894173 (+ 36us) client_negotiation.cc:240] Negotiation successful
0806 01:00:56.894226 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":31}
I20260806 01:00:56.894872 7189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:56.890990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:43851 (local address 127.23.242.1:41361)
0806 01:00:56.891136 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:56.891138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:56.891147 (+ 9us) server_negotiation.cc:407] Connection header received
0806 01:00:56.891180 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:56.891181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:56.891209 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:56.891432 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:00:56.891576 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.892287 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.893415 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.893892 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.894557 (+ 665us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:56.894705 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:56.894731 (+ 26us) server_negotiation.cc:299] Negotiation successful
0806 01:00:56.894800 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20260806 01:00:56.895097 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:56.890812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:43731 (local address 127.23.242.2:37291)
0806 01:00:56.890932 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:56.890935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:56.891171 (+ 236us) server_negotiation.cc:407] Connection header received
0806 01:00:56.891262 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:56.891263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:56.891279 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:56.891336 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:00:56.891575 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.892392 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.893975 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.894673 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.894756 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:56.894765 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:56.894964 (+ 199us) server_negotiation.cc:299] Negotiation successful
0806 01:00:56.895018 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 01:00:56.895156 6844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "977b8f0507154730b9186f6113edfb70" candidate_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f248ea2790943f2b843ad446336501e" is_pre_election: true
I20260806 01:00:56.895352 6844 raft_consensus.cc:2454] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f45227c50fdc415c8cb398d7f36bfdeb in term 0.
I20260806 01:00:56.895417 7188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:56.890862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:37291 (local address 127.23.242.3:43731)
0806 01:00:56.891060 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:56.891071 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:56.891191 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:56.891341 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:56.891342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:56.891344 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:00:56.891422 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:56.891424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.892524 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.892526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:56.893865 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:56.893868 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.894770 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.894772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:56.894860 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:56.894865 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 01:00:56.894933 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 01:00:56.895304 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":112,"threads_started":1}
I20260806 01:00:56.895741 7054 leader_election.cc:302] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f248ea2790943f2b843ad446336501e, f45227c50fdc415c8cb398d7f36bfdeb; no voters:
I20260806 01:00:56.895895 7186 raft_consensus.cc:2781] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 01:00:56.895874 6988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "977b8f0507154730b9186f6113edfb70" candidate_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9535377387324589a3599fe08d7eb908" is_pre_election: true
I20260806 01:00:56.895951 7186 raft_consensus.cc:482] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 01:00:56.895979 7186 raft_consensus.cc:3037] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:00:56.896040 6988 raft_consensus.cc:2454] T 977b8f0507154730b9186f6113edfb70 P 9535377387324589a3599fe08d7eb908 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f45227c50fdc415c8cb398d7f36bfdeb in term 0.
I20260806 01:00:56.897119 7186 raft_consensus.cc:504] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } }
I20260806 01:00:56.897306 7186 leader_election.cc:288] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb [CANDIDATE]: Term 1 election: Requested vote from peers 9535377387324589a3599fe08d7eb908 (127.23.242.2:37291), 9f248ea2790943f2b843ad446336501e (127.23.242.1:41361)
I20260806 01:00:56.897640 6844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "977b8f0507154730b9186f6113edfb70" candidate_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f248ea2790943f2b843ad446336501e"
I20260806 01:00:56.897722 6988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "977b8f0507154730b9186f6113edfb70" candidate_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9535377387324589a3599fe08d7eb908"
I20260806 01:00:56.898150 6988 raft_consensus.cc:3037] T 977b8f0507154730b9186f6113edfb70 P 9535377387324589a3599fe08d7eb908 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:00:56.898145 6844 raft_consensus.cc:3037] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:00:56.899691 6844 raft_consensus.cc:2454] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f45227c50fdc415c8cb398d7f36bfdeb in term 1.
I20260806 01:00:56.899691 6988 raft_consensus.cc:2454] T 977b8f0507154730b9186f6113edfb70 P 9535377387324589a3599fe08d7eb908 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f45227c50fdc415c8cb398d7f36bfdeb in term 1.
I20260806 01:00:56.900319 7056 leader_election.cc:302] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9535377387324589a3599fe08d7eb908, f45227c50fdc415c8cb398d7f36bfdeb; no voters:
I20260806 01:00:56.900599 7186 raft_consensus.cc:2781] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:00:56.900870 7186 raft_consensus.cc:686] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb [term 1 LEADER]: Becoming Leader. State: Replica: f45227c50fdc415c8cb398d7f36bfdeb, State: Running, Role: LEADER
I20260806 01:00:56.901153 7186 consensus_queue.cc:237] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } }
I20260806 01:00:56.902488 6699 catalog_manager.cc:5495] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb reported cstate change: term changed from 0 to 1, leader changed from <none> to f45227c50fdc415c8cb398d7f36bfdeb (127.23.242.3). New cstate: current_term: 1 leader_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } health_report { overall_health: UNKNOWN } } }
W20260806 01:00:56.910260 7170 tablet.cc:2369] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 01:00:56.929908 7180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:56.926949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49928 (local address 127.23.242.3:34067)
0806 01:00:56.927145 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:56.927149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:56.927165 (+ 16us) server_negotiation.cc:407] Connection header received
0806 01:00:56.927223 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:56.927225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:56.927255 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:56.927675 (+ 420us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 01:00:56.927825 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.928305 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.929032 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:56.929228 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:56.929312 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:56.929516 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 01:00:56.929632 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 01:00:56.929726 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:56.929752 (+ 26us) server_negotiation.cc:299] Negotiation successful
0806 01:00:56.929775 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260806 01:00:56.933972 6844 raft_consensus.cc:1261] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [term 1 FOLLOWER]: Refusing update from remote peer f45227c50fdc415c8cb398d7f36bfdeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:00:56.933972 6988 raft_consensus.cc:1261] T 977b8f0507154730b9186f6113edfb70 P 9535377387324589a3599fe08d7eb908 [term 1 FOLLOWER]: Refusing update from remote peer f45227c50fdc415c8cb398d7f36bfdeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:00:56.934441 7190 consensus_queue.cc:1035] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:00:56.934701 7186 consensus_queue.cc:1035] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:00:56.936828 7192 mvcc.cc:205] Tried to move back new op lower bound from 7315366121197899776 to 7315366121068175360. Current Snapshot: MvccSnapshot[applied={T|T < 7315366121197899776}]
I20260806 01:00:56.937332 7193 mvcc.cc:205] Tried to move back new op lower bound from 7315366121197899776 to 7315366121068175360. Current Snapshot: MvccSnapshot[applied={T|T < 7315366121197899776}]
I20260806 01:00:56.940299 6699 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 01:00:56.940410 6699 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 01:00:56.941860 6699 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875: Sending DeleteTablet for 3 replicas of tablet 977b8f0507154730b9186f6113edfb70
I20260806 01:00:56.942727 6966 tablet_service.cc:1514] Processing DeleteTablet for tablet 977b8f0507154730b9186f6113edfb70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:00:56 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:33166
I20260806 01:00:56.942790 6824 tablet_service.cc:1514] Processing DeleteTablet for tablet 977b8f0507154730b9186f6113edfb70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:00:56 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:41136
I20260806 01:00:56.942894 7101 tablet_service.cc:1514] Processing DeleteTablet for tablet 977b8f0507154730b9186f6113edfb70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:00:56 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:49918
I20260806 01:00:56.942912 7102 tablet_service.cc:1514] Processing DeleteTablet for tablet 977b8f0507154730b9186f6113edfb70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:00:56 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:49918
I20260806 01:00:56.943151 7202 tablet_replica.cc:330] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb: stopping tablet replica
I20260806 01:00:56.943264 7202 raft_consensus.cc:2229] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb [term 1 LEADER]: Raft consensus shutting down.
I20260806 01:00:56.943373 7202 raft_consensus.cc:2258] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 01:00:56.943632 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 6739
I20260806 01:00:56.945701 7202 ts_tablet_manager.cc:1905] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 01:00:56.946516 7195 mvcc.cc:205] Tried to move back new op lower bound from 7315366121197899776 to 7315366121068175360. Current Snapshot: MvccSnapshot[applied={T|T < 7315366121197899776}]
I20260806 01:00:56.948956 7202 ts_tablet_manager.cc:1918] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 01:00:56.949062 7202 log.cc:1196] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/wal/wals/977b8f0507154730b9186f6113edfb70
I20260806 01:00:56.949429 7202 ts_tablet_manager.cc:1939] T 977b8f0507154730b9186f6113edfb70 P f45227c50fdc415c8cb398d7f36bfdeb: Deleting consensus metadata
I20260806 01:00:56.950201 6684 catalog_manager.cc:4841] TS f45227c50fdc415c8cb398d7f36bfdeb (127.23.242.3:34067): tablet 977b8f0507154730b9186f6113edfb70 (table test-table [id=bb3da9ba9b454a7b93baf28f890c997a]) successfully deleted
W20260806 01:00:56.950438 6684 catalog_manager.cc:4808] TS f45227c50fdc415c8cb398d7f36bfdeb (127.23.242.3:34067): delete failed for tablet 977b8f0507154730b9186f6113edfb70 because the tablet was not found. No further retry: Not found: Tablet not found: 977b8f0507154730b9186f6113edfb70
I20260806 01:00:56.959239 7200 tablet_replica.cc:330] T 977b8f0507154730b9186f6113edfb70 P 9535377387324589a3599fe08d7eb908: stopping tablet replica
I20260806 01:00:56.959553 7200 raft_consensus.cc:2229] T 977b8f0507154730b9186f6113edfb70 P 9535377387324589a3599fe08d7eb908 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 01:00:56.959729 7200 raft_consensus.cc:2258] T 977b8f0507154730b9186f6113edfb70 P 9535377387324589a3599fe08d7eb908 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 01:00:56.960906 7200 ts_tablet_manager.cc:1905] T 977b8f0507154730b9186f6113edfb70 P 9535377387324589a3599fe08d7eb908: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 01:00:56.962680 6684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.242.1:41361 (error 108)
I20260806 01:00:56.964473 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:56.964299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:41361 (local address 127.0.0.1:41146)
0806 01:00:56.964359 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:56.964393 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.1:41361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260806 01:00:56.964571 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 6897
W20260806 01:00:56.964639 6684 catalog_manager.cc:4584] TS 9f248ea2790943f2b843ad446336501e (127.23.242.1:41361): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 977b8f0507154730b9186f6113edfb70: Network error: Client connection negotiation failed: client connection to 127.23.242.1:41361: connect: Connection refused (error 111)
I20260806 01:00:56.978782 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 7037
I20260806 01:00:56.981777 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:56.981623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:37291 (local address 127.0.0.1:33180)
0806 01:00:56.981704 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:56.981734 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.2:37291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 01:00:56.989188 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 6666
I20260806 01:00:56.997053 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:43009
--webserver_interface=127.23.242.62
--webserver_port=45829
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:41023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:43009
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:00:57.095479 7206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:00:57.095705 7206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:00:57.095732 7206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:00:57.097209 7206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:00:57.097283 7206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:00:57.097299 7206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:00:57.097312 7206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:00:57.097325 7206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:00:57.099130 7206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:43009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:43009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=45829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:00:57.099417 7206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:00:57.099738 7206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:00:57.107121 7214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:57.107126 7212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:57.107170 7211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:57.107978 7206 server_base.cc:1034] running on GCE node
I20260806 01:00:57.108408 7206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:00:57.109098 7206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:00:57.110334 7206 hybrid_clock.cc:648] HybridClock initialized: now 1785978057110320 us; error 36 us; skew 500 ppm
Aug 06 01:00:57 dist-test-slave-dhph krb5kdc[6104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978057, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:00:57.113539 7206 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 01:00:57.114066 7206 webserver.cc:460] Webserver started at http://127.23.242.62:45829/ using document root <none> and password file <none>
I20260806 01:00:57.114368 7206 fs_manager.cc:359] Metadata directory not provided
I20260806 01:00:57.114428 7206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:00:57.116083 7206 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 01:00:57.116989 7221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:57.117415 7206 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 01:00:57.117511 7206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "ef9c497be1c44b0e80cdbbb063aac875"
format_stamp: "Formatted at 2026-08-06 01:00:54 on dist-test-slave-dhph"
server_key: "ada6fa0f6ac4f6c732770b82dc5d2a25"
server_key_iv: "137db35b00a0673397254544ae372e64"
server_key_version: "encryptionkey@0"
I20260806 01:00:57.117949 7206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:00:57.139688 7206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:00:57.147042 7206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:00:57.147236 7206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:00:57.155090 7206 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:43009
I20260806 01:00:57.155529 7206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:00:57.156075 7274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:43009 every 8 connection(s)
I20260806 01:00:57.157593 7275 sys_catalog.cc:263] Verifying existing consensus state
I20260806 01:00:57.158324 7275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875: Bootstrap starting.
I20260806 01:00:57.165843 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 7206
I20260806 01:00:57.166453 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:41361
--local_ip_for_outbound_sockets=127.23.242.1
--tserver_master_addrs=127.23.242.62:43009
--webserver_port=35431
--webserver_interface=127.23.242.1
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:41023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 01:00:57.175148 7275 log.cc:824] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875: Log is configured to *not* fsync() on all Append() calls
I20260806 01:00:57.177707 7275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 01:00:57.178015 7275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875: Bootstrap complete.
I20260806 01:00:57.180619 7275 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9c497be1c44b0e80cdbbb063aac875" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43009 } }
I20260806 01:00:57.180929 7275 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef9c497be1c44b0e80cdbbb063aac875, State: Initialized, Role: FOLLOWER
I20260806 01:00:57.181181 7275 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9c497be1c44b0e80cdbbb063aac875" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43009 } }
I20260806 01:00:57.181300 7275 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:00:57.181368 7275 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:00:57.181432 7275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [term 2 FOLLOWER]: Advancing to term 3
I20260806 01:00:57.184212 7275 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9c497be1c44b0e80cdbbb063aac875" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43009 } }
I20260806 01:00:57.184389 7275 leader_election.cc:302] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef9c497be1c44b0e80cdbbb063aac875; no voters:
I20260806 01:00:57.184727 7275 leader_election.cc:288] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 01:00:57.184762 7279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 01:00:57.185025 7279 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [term 3 LEADER]: Becoming Leader. State: Replica: ef9c497be1c44b0e80cdbbb063aac875, State: Running, Role: LEADER
I20260806 01:00:57.185293 7275 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:00:57.185243 7279 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9c497be1c44b0e80cdbbb063aac875" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43009 } }
I20260806 01:00:57.187441 7281 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef9c497be1c44b0e80cdbbb063aac875. Latest consensus state: current_term: 3 leader_uuid: "ef9c497be1c44b0e80cdbbb063aac875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9c497be1c44b0e80cdbbb063aac875" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43009 } } }
I20260806 01:00:57.187566 7281 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [sys.catalog]: This master's current role is: LEADER
I20260806 01:00:57.187657 7280 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ef9c497be1c44b0e80cdbbb063aac875" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9c497be1c44b0e80cdbbb063aac875" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43009 } } }
I20260806 01:00:57.187747 7280 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9c497be1c44b0e80cdbbb063aac875 [sys.catalog]: This master's current role is: LEADER
I20260806 01:00:57.187995 7290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:00:57.188902 7290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:00:57.189069 7290 catalog_manager.cc:1259] Loaded cluster ID: 32b0a8237d7e4979ae20796c586c2c7f
I20260806 01:00:57.189200 7290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 01:00:57.189738 7290 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd308c649ad google::LogMessage::Fail() at ??:0
@ 0x7fd308c6788a google::LogMessage::SendToLog() at ??:0
@ 0x7fd308c644de google::LogMessage::Flush() at ??:0
@ 0x7fd308c65a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd310b850f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fd310b8e8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd30ab320d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd30ab27954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd30b98b6db start_thread at ??:0
@ 0x7fd30776261f clone at ??:0
W20260806 01:00:57.305028 7277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:00:57.305316 7277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:00:57.305466 7277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:00:57.307374 7277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:00:57.307482 7277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:00:57.307554 7277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 01:00:57.310763 7277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:41361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=35431
--tserver_master_addrs=127.23.242.62:43009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:00:57.311111 7277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:00:57.311378 7277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:00:57.318696 7313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:57.318711 7312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:57.318781 7315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:57.321038 7277 server_base.cc:1034] running on GCE node
I20260806 01:00:57.321658 7277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:00:57.322088 7277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:00:57.323339 7277 hybrid_clock.cc:648] HybridClock initialized: now 1785978057323295 us; error 59 us; skew 500 ppm
Aug 06 01:00:57 dist-test-slave-dhph krb5kdc[6104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978057, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:00:57.327533 7277 init.cc:377] Logged in from keytab as kudu/127.23.242.1@KRBTEST.COM (short username kudu)
I20260806 01:00:57.328455 7277 webserver.cc:460] Webserver started at http://127.23.242.1:35431/ using document root <none> and password file <none>
I20260806 01:00:57.328833 7277 fs_manager.cc:359] Metadata directory not provided
I20260806 01:00:57.328931 7277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:00:57.330788 7277 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260806 01:00:57.332031 7322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:57.332444 7277 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:00:57.332587 7277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "9f248ea2790943f2b843ad446336501e"
format_stamp: "Formatted at 2026-08-06 01:00:55 on dist-test-slave-dhph"
server_key: "2a2d20195556cab78e89075edfc29feb"
server_key_iv: "3d364d3cd187e0345f3d48a77cc70c9b"
server_key_version: "encryptionkey@0"
I20260806 01:00:57.333141 7277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:00:57.349925 7277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:00:57.353473 7277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:00:57.353667 7277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:00:57.353973 7277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:00:57.354776 7329 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 01:00:57.356287 7277 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 01:00:57.356348 7277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260806 01:00:57.356382 7277 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 01:00:57.357285 7277 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 01:00:57.357334 7277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260806 01:00:57.357467 7329 tablet_bootstrap.cc:492] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e: Bootstrap starting.
I20260806 01:00:57.365326 7277 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:41361
I20260806 01:00:57.365562 7436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:41361 every 8 connection(s)
I20260806 01:00:57.365927 7277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 01:00:57.371110 7329 log.cc:824] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e: Log is configured to *not* fsync() on all Append() calls
I20260806 01:00:57.374161 7329 tablet_bootstrap.cc:492] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 01:00:57.374503 7329 tablet_bootstrap.cc:492] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e: Bootstrap complete.
I20260806 01:00:57.375140 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 7277
I20260806 01:00:57.375439 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:37291
--local_ip_for_outbound_sockets=127.23.242.2
--tserver_master_addrs=127.23.242.62:43009
--webserver_port=36743
--webserver_interface=127.23.242.2
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:41023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 01:00:57.375990 7329 ts_tablet_manager.cc:1397] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.010s
I20260806 01:00:57.377785 7329 raft_consensus.cc:348] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } }
I20260806 01:00:57.378044 7329 raft_consensus.cc:729] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f248ea2790943f2b843ad446336501e, State: Initialized, Role: FOLLOWER
I20260806 01:00:57.378235 7329 consensus_queue.cc:260] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } }
I20260806 01:00:57.378553 7329 ts_tablet_manager.cc:1428] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
W20260806 01:00:57.450876 7439 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:00:57.366874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.1:35967)
0806 01:00:57.367115 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:57.367461 (+ 346us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:57.368385 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:57.450613 (+ 82228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":112,"threads_started":1}
W20260806 01:00:57.451283 7437 heartbeater.cc:644] Failed to heartbeat to 127.23.242.62:43009 (0 consecutive failures): Network error: Failed to ping master at 127.23.242.62:43009: Client connection negotiation failed: client connection to 127.23.242.62:43009: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 01:00:57.451768 7439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:57.451627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.1:32995)
0806 01:00:57.451672 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:57.451739 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260806 01:00:57.452260 7439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:57.452118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.1:42633)
0806 01:00:57.452204 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:57.452237 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260806 01:00:57.452456 7437 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 01:00:57.476387 7442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:00:57.476713 7442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:00:57.476775 7442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:00:57.478076 7442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:00:57.478152 7442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:00:57.478202 7442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 01:00:57.479835 7442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:37291
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=36743
--tserver_master_addrs=127.23.242.62:43009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:00:57.480149 7442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:00:57.480406 7442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:00:57.486343 7452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:57.486371 7450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:57.486531 7442 server_base.cc:1034] running on GCE node
W20260806 01:00:57.486433 7449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:57.486889 7442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:00:57.487116 7442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:00:57.488303 7442 hybrid_clock.cc:648] HybridClock initialized: now 1785978057488289 us; error 48 us; skew 500 ppm
Aug 06 01:00:57 dist-test-slave-dhph krb5kdc[6104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978057, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:00:57.491079 7442 init.cc:377] Logged in from keytab as kudu/127.23.242.2@KRBTEST.COM (short username kudu)
I20260806 01:00:57.491689 7442 webserver.cc:460] Webserver started at http://127.23.242.2:36743/ using document root <none> and password file <none>
I20260806 01:00:57.491935 7442 fs_manager.cc:359] Metadata directory not provided
I20260806 01:00:57.492007 7442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:00:57.493496 7442 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:00:57.494283 7459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:57.494499 7442 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:00:57.494648 7442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "9535377387324589a3599fe08d7eb908"
format_stamp: "Formatted at 2026-08-06 01:00:55 on dist-test-slave-dhph"
server_key: "2162ed441f203abeccede0ce1e71fcf2"
server_key_iv: "87a15761a8ec33827ac3242d36405cf0"
server_key_version: "encryptionkey@0"
I20260806 01:00:57.495144 7442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:00:57.517948 7442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:00:57.520979 7442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:00:57.521217 7442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:00:57.521538 7442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:00:57.522192 7466 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260806 01:00:57.524063 7466 ts_tablet_manager.cc:1852] T 977b8f0507154730b9186f6113edfb70 P 9535377387324589a3599fe08d7eb908: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260806 01:00:57.524379 7466 ts_tablet_manager.cc:1905] T 977b8f0507154730b9186f6113edfb70 P 9535377387324589a3599fe08d7eb908: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 01:00:57.526810 7466 ts_tablet_manager.cc:1918] T 977b8f0507154730b9186f6113edfb70 P 9535377387324589a3599fe08d7eb908: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 01:00:57.526918 7466 log.cc:1196] T 977b8f0507154730b9186f6113edfb70 P 9535377387324589a3599fe08d7eb908: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/wal/wals/977b8f0507154730b9186f6113edfb70
I20260806 01:00:57.527216 7466 ts_tablet_manager.cc:1939] T 977b8f0507154730b9186f6113edfb70 P 9535377387324589a3599fe08d7eb908: Deleting consensus metadata
I20260806 01:00:57.527468 7442 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260806 01:00:57.527547 7442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260806 01:00:57.527598 7442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:00:57.527652 7442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:57.535506 7442 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:37291
I20260806 01:00:57.535624 7576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:37291 every 8 connection(s)
I20260806 01:00:57.536083 7442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 01:00:57.537422 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:57.536833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.2:38309)
0806 01:00:57.537029 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:57.537106 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":71,"threads_started":1}
W20260806 01:00:57.537789 7577 heartbeater.cc:644] Failed to heartbeat to 127.23.242.62:43009 (0 consecutive failures): Network error: Failed to ping master at 127.23.242.62:43009: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
I20260806 01:00:57.538178 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:57.537999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.2:33255)
0806 01:00:57.538117 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:57.538155 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260806 01:00:57.538776 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:57.538539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.2:59755)
0806 01:00:57.538613 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:57.538720 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260806 01:00:57.538885 7577 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 01:00:57.543515 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 7442
I20260806 01:00:57.543802 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:34067
--local_ip_for_outbound_sockets=127.23.242.3
--tserver_master_addrs=127.23.242.62:43009
--webserver_port=46735
--webserver_interface=127.23.242.3
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:41023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:00:57.642984 7580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:00:57.643275 7580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:00:57.643321 7580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:00:57.644591 7580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:00:57.644670 7580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:00:57.644719 7580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 01:00:57.646550 7580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:41023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:34067
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=46735
--tserver_master_addrs=127.23.242.62:43009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:00:57.647006 7580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:00:57.647367 7580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:00:57.653901 7586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:57.654012 7580 server_base.cc:1034] running on GCE node
W20260806 01:00:57.653921 7588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:00:57.653905 7585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:00:57.654706 7580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:00:57.655074 7580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:00:57.656250 7580 hybrid_clock.cc:648] HybridClock initialized: now 1785978057656256 us; error 49 us; skew 500 ppm
Aug 06 01:00:57 dist-test-slave-dhph krb5kdc[6104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978057, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:00:57.659399 7580 init.cc:377] Logged in from keytab as kudu/127.23.242.3@KRBTEST.COM (short username kudu)
I20260806 01:00:57.659979 7580 webserver.cc:460] Webserver started at http://127.23.242.3:46735/ using document root <none> and password file <none>
I20260806 01:00:57.660270 7580 fs_manager.cc:359] Metadata directory not provided
I20260806 01:00:57.660348 7580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:00:57.661922 7580 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:00:57.662899 7595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:57.663415 7580 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:00:57.663561 7580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "f45227c50fdc415c8cb398d7f36bfdeb"
format_stamp: "Formatted at 2026-08-06 01:00:55 on dist-test-slave-dhph"
server_key: "2c9d7693ce1ef7f0243182d01de82540"
server_key_iv: "8a151c8ca02db6200627325899c2b267"
server_key_version: "encryptionkey@0"
I20260806 01:00:57.664100 7580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:00:57.675505 7580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:00:57.677889 7580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:00:57.678081 7580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:00:57.678342 7580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:00:57.678740 7580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:00:57.678795 7580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:57.678838 7580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:00:57.678864 7580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:00:57.685917 7580 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:34067
I20260806 01:00:57.685963 7710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:34067 every 8 connection(s)
I20260806 01:00:57.686368 7580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
I20260806 01:00:57.687623 7713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:57.687229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.3:53155)
0806 01:00:57.687395 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:57.687477 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
W20260806 01:00:57.688186 7711 heartbeater.cc:644] Failed to heartbeat to 127.23.242.62:43009 (0 consecutive failures): Network error: Failed to ping master at 127.23.242.62:43009: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
I20260806 01:00:57.688519 7713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:57.688413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.3:41681)
0806 01:00:57.688465 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:57.688497 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260806 01:00:57.688820 7713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:57.688725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.3:50659)
0806 01:00:57.688774 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:57.688804 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260806 01:00:57.689038 7711 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 01:00:57.691157 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 7580
I20260806 01:00:58.453291 7715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:58.452895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.1:47079)
0806 01:00:58.453107 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:58.453222 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":68,"threads_started":1}
I20260806 01:00:58.539546 7716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:58.539313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.2:60507)
0806 01:00:58.539461 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:58.539503 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":74,"threads_started":1}
I20260806 01:00:58.689750 7717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:58.689466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.3:47915)
0806 01:00:58.689641 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:58.689686 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":70,"threads_started":1}
I20260806 01:00:58.978484 7718 raft_consensus.cc:482] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 01:00:58.978850 7718 raft_consensus.cc:504] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } }
I20260806 01:00:58.979233 7718 leader_election.cc:288] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f45227c50fdc415c8cb398d7f36bfdeb (127.23.242.3:34067), 9535377387324589a3599fe08d7eb908 (127.23.242.2:37291)
Aug 06 01:00:58 dist-test-slave-dhph krb5kdc[6104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978057, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for kudu/127.23.242.3@KRBTEST.COM
Aug 06 01:00:58 dist-test-slave-dhph krb5kdc[6104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978057, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for kudu/127.23.242.2@KRBTEST.COM
I20260806 01:00:58.991461 7721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:58.979687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:37351 (local address 127.23.242.2:37291)
0806 01:00:58.979862 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:58.979876 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:58.979901 (+ 25us) server_negotiation.cc:407] Connection header received
0806 01:00:58.980548 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:58.980551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:58.980613 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:58.980751 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:58.981490 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:58.982719 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:58.983485 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:58.983919 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:58.984052 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:58.987299 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:58.987303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:58.987303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:58.987313 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:58.989309 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:58.989923 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:58.989927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:58.989928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:58.990036 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:58.990339 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:58.990340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:58.990340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:58.990701 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:58.990812 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:58.991136 (+ 324us) server_negotiation.cc:299] Negotiation successful
0806 01:00:58.991329 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":55,"threads_started":1}
I20260806 01:00:58.991575 7720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:58.979499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:34067 (local address 127.23.242.1:43521)
0806 01:00:58.979768 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:58.979781 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:58.979847 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:58.980884 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:58.980885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:58.981169 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:00:58.981346 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:58.981351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:58.982574 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:58.982575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:58.983346 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:58.983348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:58.983959 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:58.983960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:58.984070 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:58.984765 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:00:58.984773 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:00:58.986935 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:00:58.989565 (+ 2630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:58.989566 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:58.989566 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:58.989854 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:58.990233 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:58.990234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:58.990235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:58.990305 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:58.990981 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:00:58.990982 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:00:58.991109 (+ 127us) client_negotiation.cc:769] Sending connection context
0806 01:00:58.991185 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 01:00:58.991385 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":65,"thread_start_us":166,"threads_started":1}
I20260806 01:00:58.991591 7722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:58.979747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:43521 (local address 127.23.242.3:34067)
0806 01:00:58.979917 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 01:00:58.979925 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 01:00:58.979953 (+ 28us) server_negotiation.cc:407] Connection header received
0806 01:00:58.980532 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:00:58.980535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:00:58.980596 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:00:58.980751 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:00:58.981462 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:58.982478 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:58.983452 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:00:58.983851 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:00:58.983982 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:58.987066 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:00:58.987072 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:00:58.987073 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:00:58.987086 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:00:58.989447 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:58.990021 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:58.990024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:58.990025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:58.990136 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:00:58.990466 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:00:58.990467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:00:58.990468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:00:58.990882 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:00:58.990985 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 01:00:58.991278 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 01:00:58.991428 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":65,"threads_started":1}
I20260806 01:00:58.991458 7719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:58.979479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:37291 (local address 127.23.242.1:37351)
0806 01:00:58.979717 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:58.979730 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 01:00:58.979795 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:00:58.980836 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:00:58.980839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:00:58.981205 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:00:58.981347 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:58.981352 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:58.982817 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:58.982818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:58.983379 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:00:58.983380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:00:58.984009 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:00:58.984010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:00:58.984087 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:00:58.984872 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:00:58.984877 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:00:58.987107 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:00:58.989416 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:58.989427 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:58.989429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:58.989818 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:58.990134 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:00:58.990135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:00:58.990135 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:00:58.990256 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:00:58.990803 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:00:58.990811 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:00:58.990948 (+ 137us) client_negotiation.cc:769] Sending connection context
0806 01:00:58.991087 (+ 139us) client_negotiation.cc:240] Negotiation successful
0806 01:00:58.991302 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":4,"thread_start_us":50,"threads_started":1}
W20260806 01:00:58.992602 7324 leader_election.cc:341] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f45227c50fdc415c8cb398d7f36bfdeb (127.23.242.3:34067): Not found: Tablet not found: 977b8f0507154730b9186f6113edfb70
W20260806 01:00:58.993201 7326 leader_election.cc:341] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9535377387324589a3599fe08d7eb908 (127.23.242.2:37291): Not found: Tablet not found: 977b8f0507154730b9186f6113edfb70
I20260806 01:00:58.993268 7326 leader_election.cc:302] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f248ea2790943f2b843ad446336501e; no voters: 9535377387324589a3599fe08d7eb908, f45227c50fdc415c8cb398d7f36bfdeb
I20260806 01:00:58.993387 7718 raft_consensus.cc:2726] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 01:00:59.454416 7719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:59.454170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.1:52565)
0806 01:00:59.454295 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:59.454359 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260806 01:00:59.540800 7723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:59.540478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.2:59829)
0806 01:00:59.540648 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:59.540714 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":96,"threads_started":1}
I20260806 01:00:59.690696 7724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:00:59.690354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.3:51385)
0806 01:00:59.690540 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 01:00:59.690638 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":76,"threads_started":1}
I20260806 01:01:00.455587 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:00.455211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.1:59731)
0806 01:01:00.455439 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:00.455507 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":103,"threads_started":1}
I20260806 01:01:00.541673 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:00.541368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.2:47467)
0806 01:01:00.541534 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:00.541584 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":65,"threads_started":1}
I20260806 01:01:00.691651 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:00.691347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.3:60405)
0806 01:01:00.691513 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:00.691560 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":74,"threads_started":1}
I20260806 01:01:00.784545 7728 raft_consensus.cc:482] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 01:01:00.784658 7728 raft_consensus.cc:504] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } }
I20260806 01:01:00.785009 7728 leader_election.cc:288] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f45227c50fdc415c8cb398d7f36bfdeb (127.23.242.3:34067), 9535377387324589a3599fe08d7eb908 (127.23.242.2:37291)
W20260806 01:01:00.785395 7326 leader_election.cc:341] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9535377387324589a3599fe08d7eb908 (127.23.242.2:37291): Not found: Tablet not found: 977b8f0507154730b9186f6113edfb70
W20260806 01:01:00.785468 7324 leader_election.cc:341] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f45227c50fdc415c8cb398d7f36bfdeb (127.23.242.3:34067): Not found: Tablet not found: 977b8f0507154730b9186f6113edfb70
I20260806 01:01:00.785526 7324 leader_election.cc:302] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f248ea2790943f2b843ad446336501e; no voters: 9535377387324589a3599fe08d7eb908, f45227c50fdc415c8cb398d7f36bfdeb
I20260806 01:01:00.785634 7728 raft_consensus.cc:2726] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 01:01:01.456807 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:01.456313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.1:40931)
0806 01:01:01.456676 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:01.456743 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":128,"threads_started":1}
I20260806 01:01:01.542683 7730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:01.542338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.2:39169)
0806 01:01:01.542505 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:01.542554 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":77,"threads_started":1}
I20260806 01:01:01.692837 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:01.692524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.3:58371)
0806 01:01:01.692746 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:01.692788 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":114,"threads_started":1}
I20260806 01:01:02.457783 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:02.457509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.1:46087)
0806 01:01:02.457678 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:02.457733 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":69,"threads_started":1}
I20260806 01:01:02.543642 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:02.543272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.2:41283)
0806 01:01:02.543431 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:02.543481 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":68,"threads_started":1}
I20260806 01:01:02.693912 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:02.693559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.3:54057)
0806 01:01:02.693755 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:02.693861 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":93,"threads_started":1}
I20260806 01:01:03.116730 7735 raft_consensus.cc:482] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 01:01:03.116899 7735 raft_consensus.cc:504] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } }
I20260806 01:01:03.117208 7735 leader_election.cc:288] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f45227c50fdc415c8cb398d7f36bfdeb (127.23.242.3:34067), 9535377387324589a3599fe08d7eb908 (127.23.242.2:37291)
W20260806 01:01:03.118089 7326 leader_election.cc:341] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9535377387324589a3599fe08d7eb908 (127.23.242.2:37291): Not found: Tablet not found: 977b8f0507154730b9186f6113edfb70
W20260806 01:01:03.118264 7324 leader_election.cc:341] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f45227c50fdc415c8cb398d7f36bfdeb (127.23.242.3:34067): Not found: Tablet not found: 977b8f0507154730b9186f6113edfb70
I20260806 01:01:03.118376 7324 leader_election.cc:302] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f248ea2790943f2b843ad446336501e; no voters: 9535377387324589a3599fe08d7eb908, f45227c50fdc415c8cb398d7f36bfdeb
I20260806 01:01:03.118479 7735 raft_consensus.cc:2726] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 01:01:03.458935 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:03.458500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.1:42207)
0806 01:01:03.458831 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:03.458881 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":235,"threads_started":1}
I20260806 01:01:03.544663 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:03.544361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.2:56185)
0806 01:01:03.544558 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:03.544611 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":91,"threads_started":1}
I20260806 01:01:03.695282 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:03.694944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.3:33901)
0806 01:01:03.695171 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:03.695224 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":76,"threads_started":1}
I20260806 01:01:04.460084 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:04.459792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.1:49371)
0806 01:01:04.459971 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:04.460032 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":86,"threads_started":1}
I20260806 01:01:04.545795 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:04.545405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.2:50157)
0806 01:01:04.545644 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:04.545723 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":76,"threads_started":1}
I20260806 01:01:04.696950 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:04.696612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.3:35361)
0806 01:01:04.696830 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:04.696888 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":99,"threads_started":1}
I20260806 01:01:05.461135 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:05.460796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.1:49407)
0806 01:01:05.461024 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:05.461073 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":132,"threads_started":1}
I20260806 01:01:05.546774 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:05.546488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.2:60169)
0806 01:01:05.546676 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:05.546722 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":68,"threads_started":1}
I20260806 01:01:05.697908 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:05.697552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.3:58695)
0806 01:01:05.697800 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:05.697857 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":105,"threads_started":1}
I20260806 01:01:06.462229 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:06.461939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.1:39603)
0806 01:01:06.462124 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:06.462175 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":73,"threads_started":1}
I20260806 01:01:06.547708 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:06.547396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.2:37679)
0806 01:01:06.547615 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:06.547660 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":96,"threads_started":1}
I20260806 01:01:06.652026 7747 raft_consensus.cc:482] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 01:01:06.652138 7747 raft_consensus.cc:504] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } }
I20260806 01:01:06.652384 7747 leader_election.cc:288] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f45227c50fdc415c8cb398d7f36bfdeb (127.23.242.3:34067), 9535377387324589a3599fe08d7eb908 (127.23.242.2:37291)
W20260806 01:01:06.652906 7324 leader_election.cc:341] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f45227c50fdc415c8cb398d7f36bfdeb (127.23.242.3:34067): Not found: Tablet not found: 977b8f0507154730b9186f6113edfb70
W20260806 01:01:06.653205 7326 leader_election.cc:341] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9535377387324589a3599fe08d7eb908 (127.23.242.2:37291): Not found: Tablet not found: 977b8f0507154730b9186f6113edfb70
I20260806 01:01:06.653267 7326 leader_election.cc:302] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f248ea2790943f2b843ad446336501e; no voters: 9535377387324589a3599fe08d7eb908, f45227c50fdc415c8cb398d7f36bfdeb
I20260806 01:01:06.653388 7747 raft_consensus.cc:2726] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 01:01:06.698798 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:06.698517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.3:33311)
0806 01:01:06.698679 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:06.698734 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":98,"threads_started":1}
I20260806 01:01:07.463344 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:07.463044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.1:45953)
0806 01:01:07.463240 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:07.463293 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":68,"threads_started":1}
I20260806 01:01:07.548650 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:07.548359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.2:34469)
0806 01:01:07.548542 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:07.548597 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":75,"threads_started":1}
I20260806 01:01:07.699822 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:07.699533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.3:48761)
0806 01:01:07.699719 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:07.699770 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":73,"threads_started":1}
I20260806 01:01:08.464300 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:08.463993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.1:37369)
0806 01:01:08.464199 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:08.464250 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":81,"threads_started":1}
I20260806 01:01:08.549615 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:08.549312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.2:48949)
0806 01:01:08.549471 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:08.549516 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":71,"threads_started":1}
I20260806 01:01:08.700747 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:08.700489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.3:46681)
0806 01:01:08.700652 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:08.700695 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":66,"threads_started":1}
I20260806 01:01:09.465253 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:09.464988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.1:35327)
0806 01:01:09.465149 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:09.465199 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":77,"threads_started":1}
I20260806 01:01:09.550848 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:09.550455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.2:48341)
0806 01:01:09.550749 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:09.550795 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":82,"threads_started":1}
I20260806 01:01:09.701561 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:09.701320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.3:48083)
0806 01:01:09.701474 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:09.701514 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":67,"threads_started":1}
I20260806 01:01:10.466207 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:10.465924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.1:46767)
0806 01:01:10.466088 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:10.466154 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20260806 01:01:10.551792 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:10.551435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.2:57663)
0806 01:01:10.551699 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:10.551741 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":168,"threads_started":1}
I20260806 01:01:10.702441 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:10.702196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.3:43783)
0806 01:01:10.702343 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:10.702391 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20260806 01:01:11.467396 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:11.467061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.1:38263)
0806 01:01:11.467288 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:11.467346 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":68,"threads_started":1}
I20260806 01:01:11.552793 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:11.552488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.2:35817)
0806 01:01:11.552686 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:11.552739 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260806 01:01:11.703439 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:11.703132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.3:42357)
0806 01:01:11.703339 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:11.703386 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":90,"threads_started":1}
I20260806 01:01:12.468452 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:12.468200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.1:45407)
0806 01:01:12.468359 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:12.468404 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":66,"threads_started":1}
I20260806 01:01:12.554149 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:12.553790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.2:60127)
0806 01:01:12.554027 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:12.554093 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":93,"threads_started":1}
I20260806 01:01:12.674209 7767 raft_consensus.cc:482] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 01:01:12.674347 7767 raft_consensus.cc:504] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45227c50fdc415c8cb398d7f36bfdeb" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 34067 } } peers { permanent_uuid: "9535377387324589a3599fe08d7eb908" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 37291 } } peers { permanent_uuid: "9f248ea2790943f2b843ad446336501e" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41361 } }
I20260806 01:01:12.674623 7767 leader_election.cc:288] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f45227c50fdc415c8cb398d7f36bfdeb (127.23.242.3:34067), 9535377387324589a3599fe08d7eb908 (127.23.242.2:37291)
W20260806 01:01:12.675343 7326 leader_election.cc:341] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9535377387324589a3599fe08d7eb908 (127.23.242.2:37291): Not found: Tablet not found: 977b8f0507154730b9186f6113edfb70
W20260806 01:01:12.675441 7324 leader_election.cc:341] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f45227c50fdc415c8cb398d7f36bfdeb (127.23.242.3:34067): Not found: Tablet not found: 977b8f0507154730b9186f6113edfb70
I20260806 01:01:12.675485 7324 leader_election.cc:302] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f248ea2790943f2b843ad446336501e; no voters: 9535377387324589a3599fe08d7eb908, f45227c50fdc415c8cb398d7f36bfdeb
I20260806 01:01:12.675575 7767 raft_consensus.cc:2726] T 977b8f0507154730b9186f6113edfb70 P 9f248ea2790943f2b843ad446336501e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 01:01:12.701330 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 7277
I20260806 01:01:12.704442 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:12.704182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43009 (local address 127.23.242.3:56549)
0806 01:01:12.704324 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:12.704375 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":60,"threads_started":1}
I20260806 01:01:12.708612 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 7442
I20260806 01:01:12.715879 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 7580
2026-08-06T01:01:12Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19327 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 01:01:12 dist-test-slave-dhph krb5kdc[7772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 01:01:12 dist-test-slave-dhph krb5kdc[7772](info): set up 2 sockets
Aug 06 01:01:12 dist-test-slave-dhph krb5kdc[7772](info): commencing operation
krb5kdc: starting...
W20260806 01:01:13.907024 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.167s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 01:01:14 dist-test-slave-dhph krb5kdc[7772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T01:01:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:01:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 01:01:14.088290 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:40935
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:40935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:01:14.182066 7785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:01:14.182293 7785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:01:14.182317 7785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:01:14.183703 7785 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 01:01:14.183763 7785 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 01:01:14.183776 7785 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 01:01:14.183789 7785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:01:14.183802 7785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:01:14.183815 7785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:01:14.183887 7785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:01:14.183908 7785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:01:14.185504 7785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:40935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:40935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:01:14.185757 7785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:01:14.186034 7785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:01:14.192233 7790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:01:14.192310 7793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:01:14.192613 7785 server_base.cc:1034] running on GCE node
W20260806 01:01:14.192698 7791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:01:14.192989 7785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:01:14.193300 7785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:01:14.194474 7785 hybrid_clock.cc:648] HybridClock initialized: now 1785978074194423 us; error 66 us; skew 500 ppm
Aug 06 01:01:14 dist-test-slave-dhph krb5kdc[7772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978074, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:01:14.207986 7785 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 01:01:14.208591 7785 webserver.cc:460] Webserver started at http://127.23.242.62:35029/ using document root <none> and password file <none>
I20260806 01:01:14.208904 7785 fs_manager.cc:359] Metadata directory not provided
I20260806 01:01:14.208961 7785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:01:14.209122 7785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:01:14.210384 7785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "9e2f2c35989b42cd9f890a5776564bc5"
format_stamp: "Formatted at 2026-08-06 01:01:14 on dist-test-slave-dhph"
server_key: "9b3a0bba63de6a206fb1c8e231cbdf4b"
server_key_iv: "3139f1fd11aaefb7e7ed956a0bf4327b"
server_key_version: "encryptionkey@0"
I20260806 01:01:14.210872 7785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "9e2f2c35989b42cd9f890a5776564bc5"
format_stamp: "Formatted at 2026-08-06 01:01:14 on dist-test-slave-dhph"
server_key: "9b3a0bba63de6a206fb1c8e231cbdf4b"
server_key_iv: "3139f1fd11aaefb7e7ed956a0bf4327b"
server_key_version: "encryptionkey@0"
I20260806 01:01:14.212651 7785 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260806 01:01:14.213740 7800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:01:14.214273 7785 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 01:01:14.214460 7785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "9e2f2c35989b42cd9f890a5776564bc5"
format_stamp: "Formatted at 2026-08-06 01:01:14 on dist-test-slave-dhph"
server_key: "9b3a0bba63de6a206fb1c8e231cbdf4b"
server_key_iv: "3139f1fd11aaefb7e7ed956a0bf4327b"
server_key_version: "encryptionkey@0"
I20260806 01:01:14.214593 7785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:01:14.239245 7785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:01:14.242293 7785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:01:14.242542 7785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:01:14.250667 7785 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:40935
I20260806 01:01:14.250741 7853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:40935 every 8 connection(s)
I20260806 01:01:14.251209 7785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:01:14.252324 7854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:01:14.255084 7854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5: Bootstrap starting.
I20260806 01:01:14.255826 7854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5: Neither blocks nor log segments found. Creating new log.
I20260806 01:01:14.256222 7854 log.cc:824] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5: Log is configured to *not* fsync() on all Append() calls
I20260806 01:01:14.256390 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 7785
I20260806 01:01:14.256533 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:01:14.256884 24520 external_mini_cluster.cc:1428] Setting key b110219049f4400a459be2c81be1f561
I20260806 01:01:14.257148 7854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5: No bootstrap required, opened a new log
I20260806 01:01:14.262120 7854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2f2c35989b42cd9f890a5776564bc5" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 40935 } }
I20260806 01:01:14.262439 7854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:01:14.262494 7854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e2f2c35989b42cd9f890a5776564bc5, State: Initialized, Role: FOLLOWER
I20260806 01:01:14.262717 7854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2f2c35989b42cd9f890a5776564bc5" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 40935 } }
I20260806 01:01:14.262816 7854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:01:14.262853 7854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:01:14.263023 7854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:01:14.264726 7854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2f2c35989b42cd9f890a5776564bc5" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 40935 } }
I20260806 01:01:14.264896 7854 leader_election.cc:302] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e2f2c35989b42cd9f890a5776564bc5; no voters:
I20260806 01:01:14.265127 7854 leader_election.cc:288] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:01:14.265264 7859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:01:14.265461 7859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [term 1 LEADER]: Becoming Leader. State: Replica: 9e2f2c35989b42cd9f890a5776564bc5, State: Running, Role: LEADER
I20260806 01:01:14.265496 7854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:01:14.265556 7859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2f2c35989b42cd9f890a5776564bc5" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 40935 } }
I20260806 01:01:14.266748 7860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e2f2c35989b42cd9f890a5776564bc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2f2c35989b42cd9f890a5776564bc5" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 40935 } } }
I20260806 01:01:14.266876 7860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [sys.catalog]: This master's current role is: LEADER
I20260806 01:01:14.267287 7861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e2f2c35989b42cd9f890a5776564bc5. Latest consensus state: current_term: 1 leader_uuid: "9e2f2c35989b42cd9f890a5776564bc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2f2c35989b42cd9f890a5776564bc5" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 40935 } } }
I20260806 01:01:14.267397 7861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [sys.catalog]: This master's current role is: LEADER
W20260806 01:01:14.268414 7874 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 01:01:14.268543 7874 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 06 01:01:14 dist-test-slave-dhph krb5kdc[7772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:01:14.268664 7872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:01:14.268971 7872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:01:14.271071 7872 catalog_manager.cc:1347] Generated new cluster ID: 3128c701d3a842a3b698fe3eccd0f8db
I20260806 01:01:14.271203 7872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:01:14.274307 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:14.257546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41086 (local address 127.23.242.62:40935)
0806 01:01:14.257884 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0806 01:01:14.257895 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 01:01:14.257931 (+ 36us) server_negotiation.cc:407] Connection header received
0806 01:01:14.259020 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:01:14.259031 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:01:14.259107 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:01:14.259748 (+ 641us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:01:14.260632 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.262286 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.263865 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.264629 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.264806 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:14.269386 (+ 4580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:01:14.269391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:01:14.269391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:01:14.269406 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:01:14.272204 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:01:14.272696 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:01:14.272700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:01:14.272700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:01:14.272834 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:01:14.273079 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:01:14.273080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:01:14.273081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:01:14.273451 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:01:14.273643 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 01:01:14.273766 (+ 123us) server_negotiation.cc:299] Negotiation successful
0806 01:01:14.273931 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":120,"threads_started":1}
I20260806 01:01:14.303191 7872 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:01:14.304605 7872 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:01:14.308131 7872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5: Generated new TSK 0
I20260806 01:01:14.308418 7872 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.242.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
I20260806 01:01:14.352916 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:0
--local_ip_for_outbound_sockets=127.23.242.1
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:40935
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:01:14.461787 7880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:01:14.462090 7880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:01:14.462142 7880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:01:14.463799 7880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:01:14.463893 7880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:01:14.463954 7880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 01:01:14.466321 7880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:40935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:01:14.466742 7880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:01:14.467047 7880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:01:14.473776 7888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:01:14.473861 7885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:01:14.473872 7886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:01:14.473920 7880 server_base.cc:1034] running on GCE node
I20260806 01:01:14.474279 7880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:01:14.474551 7880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:01:14.475778 7880 hybrid_clock.cc:648] HybridClock initialized: now 1785978074475761 us; error 34 us; skew 500 ppm
Aug 06 01:01:14 dist-test-slave-dhph krb5kdc[7772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978074, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:01:14.479547 7880 init.cc:377] Logged in from keytab as kudu/127.23.242.1@KRBTEST.COM (short username kudu)
I20260806 01:01:14.480048 7880 webserver.cc:460] Webserver started at http://127.23.242.1:43115/ using document root <none> and password file <none>
I20260806 01:01:14.480315 7880 fs_manager.cc:359] Metadata directory not provided
I20260806 01:01:14.480388 7880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:01:14.480590 7880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:01:14.481691 7880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/data/instance:
uuid: "da6c6da4a5434ab3b834d8d0eee9f57c"
format_stamp: "Formatted at 2026-08-06 01:01:14 on dist-test-slave-dhph"
server_key: "1fd76abebc85df107bb8d0b8ed92a9d2"
server_key_iv: "56379cf7796a0b50a345427f1b510d64"
server_key_version: "encryptionkey@0"
I20260806 01:01:14.482066 7880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance:
uuid: "da6c6da4a5434ab3b834d8d0eee9f57c"
format_stamp: "Formatted at 2026-08-06 01:01:14 on dist-test-slave-dhph"
server_key: "1fd76abebc85df107bb8d0b8ed92a9d2"
server_key_iv: "56379cf7796a0b50a345427f1b510d64"
server_key_version: "encryptionkey@0"
I20260806 01:01:14.483767 7880 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 01:01:14.484894 7895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:01:14.485278 7880 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:01:14.485421 7880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "da6c6da4a5434ab3b834d8d0eee9f57c"
format_stamp: "Formatted at 2026-08-06 01:01:14 on dist-test-slave-dhph"
server_key: "1fd76abebc85df107bb8d0b8ed92a9d2"
server_key_iv: "56379cf7796a0b50a345427f1b510d64"
server_key_version: "encryptionkey@0"
I20260806 01:01:14.485750 7880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:01:14.518038 7880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:01:14.521278 7880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:01:14.521448 7880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:01:14.521762 7880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:01:14.522149 7880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:01:14.522213 7880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:01:14.522265 7880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:01:14.522313 7880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:01:14.530759 7880 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:43497
I20260806 01:01:14.530864 8010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:43497 every 8 connection(s)
I20260806 01:01:14.531235 7880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 01:01:14.531564 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 7880
I20260806 01:01:14.531711 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance
I20260806 01:01:14.531965 24520 external_mini_cluster.cc:1428] Setting key 35fd409496aff53a5192fa92c7b883f8
Aug 06 01:01:14 dist-test-slave-dhph krb5kdc[7772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978074, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.242.2@KRBTEST.COM; defaulting to no policy
I20260806 01:01:14.545222 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:14.532425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:49433 (local address 127.23.242.62:40935)
0806 01:01:14.532471 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0806 01:01:14.532474 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:01:14.532544 (+ 70us) server_negotiation.cc:407] Connection header received
0806 01:01:14.533353 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:01:14.533355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:01:14.533369 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:01:14.533417 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:01:14.534546 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.535440 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.536570 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.536998 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.537086 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:14.540251 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:01:14.540253 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:01:14.540253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:01:14.540265 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:01:14.542433 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:01:14.543649 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:01:14.543652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:01:14.543653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:01:14.543766 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:01:14.544236 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:01:14.544238 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:01:14.544238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:01:14.544546 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:01:14.544664 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 01:01:14.545036 (+ 372us) server_negotiation.cc:299] Negotiation successful
0806 01:01:14.545136 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 01:01:14.545446 8013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:14.532159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.1:49433)
0806 01:01:14.532430 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:14.532477 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 01:01:14.533209 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:01:14.533515 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:01:14.533521 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:01:14.534199 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:01:14.534422 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:14.534427 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.535595 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.535596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:14.536467 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:14.536469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.537148 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.537150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:14.537286 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:14.538321 (+ 1035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:01:14.538334 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:01:14.540120 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:01:14.542980 (+ 2860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:01:14.542991 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:01:14.542994 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:01:14.543319 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:01:14.543987 (+ 668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:01:14.543988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:01:14.543989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:01:14.544137 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:01:14.544701 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:01:14.544709 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:01:14.544865 (+ 156us) client_negotiation.cc:769] Sending connection context
0806 01:01:14.544979 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 01:01:14.545159 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":107,"threads_started":1}
I20260806 01:01:14.546200 8011 heartbeater.cc:347] Connected to a master server at 127.23.242.62:40935
I20260806 01:01:14.546319 8011 heartbeater.cc:464] Registering TS with master...
I20260806 01:01:14.546643 8011 heartbeater.cc:511] Master 127.23.242.62:40935 requested a full tablet report, sending...
I20260806 01:01:14.547345 7818 ts_manager.cc:194] Registered new tserver with Master: da6c6da4a5434ab3b834d8d0eee9f57c (127.23.242.1:43497)
I20260806 01:01:14.548139 7818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.1@KRBTEST.COM'} at 127.23.242.1:49433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
I20260806 01:01:14.575865 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:0
--local_ip_for_outbound_sockets=127.23.242.2
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:40935
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:01:14.687472 8016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:01:14.687747 8016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:01:14.687779 8016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:01:14.689225 8016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:01:14.689334 8016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:01:14.689380 8016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 01:01:14.691373 8016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:40935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:01:14.691653 8016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:01:14.691958 8016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:01:14.698421 8022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:01:14.698416 8024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:01:14.698412 8021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:01:14.698832 8016 server_base.cc:1034] running on GCE node
I20260806 01:01:14.699158 8016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:01:14.699471 8016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:01:14.700659 8016 hybrid_clock.cc:648] HybridClock initialized: now 1785978074700617 us; error 62 us; skew 500 ppm
Aug 06 01:01:14 dist-test-slave-dhph krb5kdc[7772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978074, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:01:14.703863 8016 init.cc:377] Logged in from keytab as kudu/127.23.242.2@KRBTEST.COM (short username kudu)
I20260806 01:01:14.704485 8016 webserver.cc:460] Webserver started at http://127.23.242.2:38511/ using document root <none> and password file <none>
I20260806 01:01:14.704764 8016 fs_manager.cc:359] Metadata directory not provided
I20260806 01:01:14.704846 8016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:01:14.705013 8016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:01:14.706203 8016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/data/instance:
uuid: "dfbb4658535f4208a4d9bdb1f31c0103"
format_stamp: "Formatted at 2026-08-06 01:01:14 on dist-test-slave-dhph"
server_key: "67f389e1ed8bdbf335c95c1873eecbb2"
server_key_iv: "1e9f00d66187292452ce4eec974ab02a"
server_key_version: "encryptionkey@0"
I20260806 01:01:14.706704 8016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance:
uuid: "dfbb4658535f4208a4d9bdb1f31c0103"
format_stamp: "Formatted at 2026-08-06 01:01:14 on dist-test-slave-dhph"
server_key: "67f389e1ed8bdbf335c95c1873eecbb2"
server_key_iv: "1e9f00d66187292452ce4eec974ab02a"
server_key_version: "encryptionkey@0"
I20260806 01:01:14.708567 8016 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260806 01:01:14.709755 8031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:01:14.710013 8016 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:01:14.710142 8016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "dfbb4658535f4208a4d9bdb1f31c0103"
format_stamp: "Formatted at 2026-08-06 01:01:14 on dist-test-slave-dhph"
server_key: "67f389e1ed8bdbf335c95c1873eecbb2"
server_key_iv: "1e9f00d66187292452ce4eec974ab02a"
server_key_version: "encryptionkey@0"
I20260806 01:01:14.710246 8016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:01:14.728665 8016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:01:14.732120 8016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:01:14.732337 8016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:01:14.732653 8016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:01:14.733069 8016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:01:14.733139 8016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:01:14.733189 8016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:01:14.733247 8016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:01:14.740772 8016 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:41497
I20260806 01:01:14.740828 8146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:41497 every 8 connection(s)
I20260806 01:01:14.741168 8016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 01:01:14.743974 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 8016
I20260806 01:01:14.744113 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance
I20260806 01:01:14.744374 24520 external_mini_cluster.cc:1428] Setting key 4dd9a3cbc7a1f1d91fe3763259c4e198
Aug 06 01:01:14 dist-test-slave-dhph krb5kdc[7772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978074, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:01:14.754123 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:14.742084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:35227 (local address 127.23.242.62:40935)
0806 01:01:14.742272 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 01:01:14.742279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 01:01:14.742295 (+ 16us) server_negotiation.cc:407] Connection header received
0806 01:01:14.742993 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:01:14.742995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:01:14.743009 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:01:14.743108 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:01:14.744310 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.744976 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.746356 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.746730 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.746826 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:14.750330 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:01:14.750332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:01:14.750333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:01:14.750347 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:01:14.752324 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:01:14.752865 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:01:14.752866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:01:14.752867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:01:14.752945 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:01:14.753305 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:01:14.753306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:01:14.753306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:01:14.753461 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:01:14.753559 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 01:01:14.753892 (+ 333us) server_negotiation.cc:299] Negotiation successful
0806 01:01:14.753998 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 01:01:14.754374 8149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:14.741978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.2:35227)
0806 01:01:14.742172 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:14.742220 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 01:01:14.742862 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:01:14.743178 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:01:14.743181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:01:14.743925 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:01:14.744194 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:14.744201 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.745140 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.745142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:14.746225 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:14.746228 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.746858 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.746860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:14.746991 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:14.747733 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:01:14.747743 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:01:14.750172 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:01:14.752428 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:01:14.752440 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:01:14.752442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:01:14.752748 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:01:14.753048 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:01:14.753049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:01:14.753050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:01:14.753201 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:01:14.753549 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:01:14.753557 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:01:14.753731 (+ 174us) client_negotiation.cc:769] Sending connection context
0806 01:01:14.753867 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 01:01:14.754073 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.23.242.3@KRBTEST.COM; defaulting to no policy
I20260806 01:01:14.755513 8147 heartbeater.cc:347] Connected to a master server at 127.23.242.62:40935
I20260806 01:01:14.755740 8147 heartbeater.cc:464] Registering TS with master...
I20260806 01:01:14.756244 8147 heartbeater.cc:511] Master 127.23.242.62:40935 requested a full tablet report, sending...
I20260806 01:01:14.756861 7818 ts_manager.cc:194] Registered new tserver with Master: dfbb4658535f4208a4d9bdb1f31c0103 (127.23.242.2:41497)
I20260806 01:01:14.757287 7818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.2@KRBTEST.COM'} at 127.23.242.2:35227
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
I20260806 01:01:14.782274 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:0
--local_ip_for_outbound_sockets=127.23.242.3
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:40935
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:01:14.888845 8152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:01:14.889156 8152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:01:14.889210 8152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:01:14.890743 8152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:01:14.890898 8152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:01:14.890955 8152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 01:01:14.892715 8152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:40935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:01:14.893028 8152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:01:14.893290 8152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:01:14.899456 8158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:01:14.899477 8157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:01:14.899453 8160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:01:14.900034 8152 server_base.cc:1034] running on GCE node
I20260806 01:01:14.900298 8152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:01:14.900588 8152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:01:14.901808 8152 hybrid_clock.cc:648] HybridClock initialized: now 1785978074901777 us; error 36 us; skew 500 ppm
Aug 06 01:01:14 dist-test-slave-dhph krb5kdc[7772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978074, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:01:14.906018 8152 init.cc:377] Logged in from keytab as kudu/127.23.242.3@KRBTEST.COM (short username kudu)
I20260806 01:01:14.907058 8152 webserver.cc:460] Webserver started at http://127.23.242.3:43959/ using document root <none> and password file <none>
I20260806 01:01:14.907359 8152 fs_manager.cc:359] Metadata directory not provided
I20260806 01:01:14.907439 8152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:01:14.907603 8152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:01:14.908963 8152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/data/instance:
uuid: "16bd3ad6217d40e19f7a6624ad9c650b"
format_stamp: "Formatted at 2026-08-06 01:01:14 on dist-test-slave-dhph"
server_key: "cc6ba1f343637f722f44cccdb65dcc64"
server_key_iv: "3f5803827a1f305886458c8f4faaf384"
server_key_version: "encryptionkey@0"
I20260806 01:01:14.909430 8152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance:
uuid: "16bd3ad6217d40e19f7a6624ad9c650b"
format_stamp: "Formatted at 2026-08-06 01:01:14 on dist-test-slave-dhph"
server_key: "cc6ba1f343637f722f44cccdb65dcc64"
server_key_iv: "3f5803827a1f305886458c8f4faaf384"
server_key_version: "encryptionkey@0"
I20260806 01:01:14.911322 8152 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260806 01:01:14.912636 8167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:01:14.913046 8152 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:01:14.913172 8152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "16bd3ad6217d40e19f7a6624ad9c650b"
format_stamp: "Formatted at 2026-08-06 01:01:14 on dist-test-slave-dhph"
server_key: "cc6ba1f343637f722f44cccdb65dcc64"
server_key_iv: "3f5803827a1f305886458c8f4faaf384"
server_key_version: "encryptionkey@0"
I20260806 01:01:14.913298 8152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:01:14.924333 8152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:01:14.927949 8152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:01:14.928108 8152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:01:14.928452 8152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:01:14.928849 8152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:01:14.928917 8152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:01:14.928972 8152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:01:14.929005 8152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:01:14.940179 8152 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:36385
I20260806 01:01:14.940240 8281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:36385 every 8 connection(s)
I20260806 01:01:14.940623 8152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
Aug 06 01:01:14 dist-test-slave-dhph krb5kdc[7772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978074, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:01:14.950083 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 8152
I20260806 01:01:14.950227 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance
I20260806 01:01:14.950510 24520 external_mini_cluster.cc:1428] Setting key e6418bd969495558056ee6e79c77e64e
I20260806 01:01:14.955281 8284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:14.941466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.3:33463)
0806 01:01:14.941736 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:14.941791 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 01:01:14.942311 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:01:14.942752 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:01:14.942756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:01:14.943893 (+ 1137us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:01:14.944137 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:14.944146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.945025 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.945026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:14.945987 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:14.945990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.946682 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.946684 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:14.946815 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:14.947909 (+ 1094us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:01:14.947922 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:01:14.950482 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:01:14.953306 (+ 2824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:01:14.953317 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:01:14.953319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:01:14.953616 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:01:14.953988 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:01:14.953988 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:01:14.953989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:01:14.954124 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:01:14.954634 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:01:14.954642 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:01:14.954804 (+ 162us) client_negotiation.cc:769] Sending connection context
0806 01:01:14.954911 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 01:01:14.955066 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":100,"threads_started":1}
I20260806 01:01:14.955318 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:14.941540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:33463 (local address 127.23.242.62:40935)
0806 01:01:14.941624 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0806 01:01:14.941626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:01:14.942029 (+ 403us) server_negotiation.cc:407] Connection header received
0806 01:01:14.942464 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:01:14.942465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:01:14.942479 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:01:14.942553 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:01:14.944271 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.944877 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.946176 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.946432 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.946566 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:14.950703 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:01:14.950705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:01:14.950706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:01:14.950718 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:01:14.953172 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:01:14.953819 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:01:14.953820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:01:14.953821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:01:14.953889 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:01:14.954292 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:01:14.954293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:01:14.954294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:01:14.954458 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:01:14.954553 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 01:01:14.954993 (+ 440us) server_negotiation.cc:299] Negotiation successful
0806 01:01:14.955241 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 01:01:14.956163 8282 heartbeater.cc:347] Connected to a master server at 127.23.242.62:40935
I20260806 01:01:14.956269 8282 heartbeater.cc:464] Registering TS with master...
I20260806 01:01:14.956486 8282 heartbeater.cc:511] Master 127.23.242.62:40935 requested a full tablet report, sending...
I20260806 01:01:14.957015 7818 ts_manager.cc:194] Registered new tserver with Master: 16bd3ad6217d40e19f7a6624ad9c650b (127.23.242.3:36385)
I20260806 01:01:14.957466 7818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.3@KRBTEST.COM'} at 127.23.242.3:33463
I20260806 01:01:14.964490 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 01:01:14.974754 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:14.966415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41092 (local address 127.23.242.62:40935)
0806 01:01:14.966496 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0806 01:01:14.966499 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:01:14.966708 (+ 209us) server_negotiation.cc:407] Connection header received
0806 01:01:14.966810 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:01:14.966811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:01:14.966822 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:01:14.966870 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:01:14.967638 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.968186 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.969181 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.969394 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.969464 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:14.971084 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:01:14.971085 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:01:14.971085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:01:14.971093 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:01:14.972651 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:01:14.973514 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:01:14.973515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:01:14.973515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:01:14.973582 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:01:14.973913 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:01:14.973914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:01:14.973915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:01:14.974100 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:01:14.974193 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 01:01:14.974403 (+ 210us) server_negotiation.cc:299] Negotiation successful
0806 01:01:14.974486 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 01:01:14.977566 7818 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 01:01:14.980176 7818 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 01:01:14.988178 8298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:14.983577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49108 (local address 127.23.242.3:36385)
0806 01:01:14.983808 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 01:01:14.983813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 01:01:14.983826 (+ 13us) server_negotiation.cc:407] Connection header received
0806 01:01:14.983985 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:01:14.983988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:01:14.984005 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:01:14.984085 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:01:14.984303 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.985362 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.986978 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.987695 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.987777 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:14.987838 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 01:01:14.988027 (+ 189us) server_negotiation.cc:299] Negotiation successful
0806 01:01:14.988081 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":90,"threads_started":1}
I20260806 01:01:14.988405 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:14.983237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:41497 (local address 127.0.0.1:33306)
0806 01:01:14.983460 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:14.983507 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 01:01:14.983583 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:01:14.984269 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:01:14.984270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:01:14.984273 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:01:14.984337 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:14.984340 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.986157 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.986157 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:14.987124 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:14.987127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.988132 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.988133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:14.988214 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:14.988218 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 01:01:14.988269 (+ 51us) client_negotiation.cc:240] Negotiation successful
0806 01:01:14.988318 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":58,"thread_start_us":151,"threads_started":1}
I20260806 01:01:14.988531 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:14.983255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:36385 (local address 127.0.0.1:49108)
0806 01:01:14.983684 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:14.983696 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 01:01:14.983845 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:01:14.984169 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:01:14.984170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:01:14.984171 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:01:14.984214 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:14.984214 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.985488 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.985489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:14.986874 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:14.986878 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.987794 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.987795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:14.987885 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:14.987901 (+ 16us) client_negotiation.cc:769] Sending connection context
0806 01:01:14.987996 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 01:01:14.988067 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":117,"threads_started":1}
I20260806 01:01:14.988690 8299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:14.983772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33306 (local address 127.23.242.2:41497)
0806 01:01:14.984003 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 01:01:14.984008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 01:01:14.984020 (+ 12us) server_negotiation.cc:407] Connection header received
0806 01:01:14.984062 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:01:14.984065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:01:14.984082 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:01:14.984182 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:01:14.984439 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.986034 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.987233 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.987969 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.988037 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:14.988095 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 01:01:14.988526 (+ 431us) server_negotiation.cc:299] Negotiation successful
0806 01:01:14.988584 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":98,"threads_started":1}
I20260806 01:01:14.988735 8294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:14.983248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:43497 (local address 127.0.0.1:56508)
0806 01:01:14.983483 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:14.983507 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 01:01:14.983583 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:01:14.984125 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:01:14.984128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:01:14.984130 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:01:14.984211 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:14.984214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.986138 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.986139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:14.987386 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:14.987389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.988448 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.988449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:14.988531 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:14.988535 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 01:01:14.988569 (+ 34us) client_negotiation.cc:240] Negotiation successful
0806 01:01:14.988629 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":57,"thread_start_us":55,"threads_started":1}
I20260806 01:01:14.989360 8297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:14.983496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56508 (local address 127.23.242.1:43497)
0806 01:01:14.983917 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0806 01:01:14.983921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 01:01:14.983932 (+ 11us) server_negotiation.cc:407] Connection header received
0806 01:01:14.983976 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:01:14.983979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:01:14.983996 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:01:14.984315 (+ 319us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:01:14.984418 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.985988 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.987482 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:14.988249 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:14.989078 (+ 829us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:14.989166 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 01:01:14.989196 (+ 30us) server_negotiation.cc:299] Negotiation successful
0806 01:01:14.989246 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":90,"threads_started":1}
I20260806 01:01:14.989363 8215 tablet_service.cc:1467] Processing CreateTablet for tablet 84609fa183f9435796e6335a7adb0b61 (DEFAULT_TABLE table=test-table [id=c6b71628fcc14282b24d0a831aee6ce6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:01:14.989446 8080 tablet_service.cc:1467] Processing CreateTablet for tablet 84609fa183f9435796e6335a7adb0b61 (DEFAULT_TABLE table=test-table [id=c6b71628fcc14282b24d0a831aee6ce6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:01:14.989785 8215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84609fa183f9435796e6335a7adb0b61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:01:14.989830 8080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84609fa183f9435796e6335a7adb0b61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:01:14.990064 7944 tablet_service.cc:1467] Processing CreateTablet for tablet 84609fa183f9435796e6335a7adb0b61 (DEFAULT_TABLE table=test-table [id=c6b71628fcc14282b24d0a831aee6ce6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:01:14.990569 7944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84609fa183f9435796e6335a7adb0b61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:01:14.992767 8301 tablet_bootstrap.cc:492] T 84609fa183f9435796e6335a7adb0b61 P 16bd3ad6217d40e19f7a6624ad9c650b: Bootstrap starting.
I20260806 01:01:14.993388 8301 tablet_bootstrap.cc:654] T 84609fa183f9435796e6335a7adb0b61 P 16bd3ad6217d40e19f7a6624ad9c650b: Neither blocks nor log segments found. Creating new log.
I20260806 01:01:14.994114 8301 log.cc:824] T 84609fa183f9435796e6335a7adb0b61 P 16bd3ad6217d40e19f7a6624ad9c650b: Log is configured to *not* fsync() on all Append() calls
I20260806 01:01:14.994889 8303 tablet_bootstrap.cc:492] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c: Bootstrap starting.
I20260806 01:01:14.995061 8301 tablet_bootstrap.cc:492] T 84609fa183f9435796e6335a7adb0b61 P 16bd3ad6217d40e19f7a6624ad9c650b: No bootstrap required, opened a new log
I20260806 01:01:14.995141 8301 ts_tablet_manager.cc:1397] T 84609fa183f9435796e6335a7adb0b61 P 16bd3ad6217d40e19f7a6624ad9c650b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 01:01:14.995384 8304 tablet_bootstrap.cc:492] T 84609fa183f9435796e6335a7adb0b61 P dfbb4658535f4208a4d9bdb1f31c0103: Bootstrap starting.
I20260806 01:01:14.995612 8303 tablet_bootstrap.cc:654] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c: Neither blocks nor log segments found. Creating new log.
I20260806 01:01:14.996166 8304 tablet_bootstrap.cc:654] T 84609fa183f9435796e6335a7adb0b61 P dfbb4658535f4208a4d9bdb1f31c0103: Neither blocks nor log segments found. Creating new log.
I20260806 01:01:14.996327 8303 log.cc:824] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c: Log is configured to *not* fsync() on all Append() calls
I20260806 01:01:14.997054 8304 log.cc:824] T 84609fa183f9435796e6335a7adb0b61 P dfbb4658535f4208a4d9bdb1f31c0103: Log is configured to *not* fsync() on all Append() calls
I20260806 01:01:14.997136 8303 tablet_bootstrap.cc:492] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c: No bootstrap required, opened a new log
I20260806 01:01:14.997203 8303 ts_tablet_manager.cc:1397] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260806 01:01:14.997668 8301 raft_consensus.cc:348] T 84609fa183f9435796e6335a7adb0b61 P 16bd3ad6217d40e19f7a6624ad9c650b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } }
I20260806 01:01:14.997879 8301 raft_consensus.cc:374] T 84609fa183f9435796e6335a7adb0b61 P 16bd3ad6217d40e19f7a6624ad9c650b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:01:14.997900 8304 tablet_bootstrap.cc:492] T 84609fa183f9435796e6335a7adb0b61 P dfbb4658535f4208a4d9bdb1f31c0103: No bootstrap required, opened a new log
I20260806 01:01:14.997922 8301 raft_consensus.cc:729] T 84609fa183f9435796e6335a7adb0b61 P 16bd3ad6217d40e19f7a6624ad9c650b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16bd3ad6217d40e19f7a6624ad9c650b, State: Initialized, Role: FOLLOWER
I20260806 01:01:14.997972 8304 ts_tablet_manager.cc:1397] T 84609fa183f9435796e6335a7adb0b61 P dfbb4658535f4208a4d9bdb1f31c0103: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 01:01:14.998093 8301 consensus_queue.cc:260] T 84609fa183f9435796e6335a7adb0b61 P 16bd3ad6217d40e19f7a6624ad9c650b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } }
I20260806 01:01:14.998435 8282 heartbeater.cc:503] Master 127.23.242.62:40935 was elected leader, sending a full tablet report...
I20260806 01:01:14.998430 8301 ts_tablet_manager.cc:1428] T 84609fa183f9435796e6335a7adb0b61 P 16bd3ad6217d40e19f7a6624ad9c650b: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260806 01:01:14.999547 8303 raft_consensus.cc:348] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } }
I20260806 01:01:14.999704 8303 raft_consensus.cc:374] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:01:14.999737 8303 raft_consensus.cc:729] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da6c6da4a5434ab3b834d8d0eee9f57c, State: Initialized, Role: FOLLOWER
I20260806 01:01:14.999919 8303 consensus_queue.cc:260] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } }
I20260806 01:01:15.000631 8011 heartbeater.cc:503] Master 127.23.242.62:40935 was elected leader, sending a full tablet report...
I20260806 01:01:15.000636 8303 ts_tablet_manager.cc:1428] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260806 01:01:15.001087 8304 raft_consensus.cc:348] T 84609fa183f9435796e6335a7adb0b61 P dfbb4658535f4208a4d9bdb1f31c0103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } }
I20260806 01:01:15.001294 8304 raft_consensus.cc:374] T 84609fa183f9435796e6335a7adb0b61 P dfbb4658535f4208a4d9bdb1f31c0103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:01:15.001350 8304 raft_consensus.cc:729] T 84609fa183f9435796e6335a7adb0b61 P dfbb4658535f4208a4d9bdb1f31c0103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfbb4658535f4208a4d9bdb1f31c0103, State: Initialized, Role: FOLLOWER
I20260806 01:01:15.001474 8304 consensus_queue.cc:260] T 84609fa183f9435796e6335a7adb0b61 P dfbb4658535f4208a4d9bdb1f31c0103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } }
I20260806 01:01:15.001860 8304 ts_tablet_manager.cc:1428] T 84609fa183f9435796e6335a7adb0b61 P dfbb4658535f4208a4d9bdb1f31c0103: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260806 01:01:15.001963 8147 heartbeater.cc:503] Master 127.23.242.62:40935 was elected leader, sending a full tablet report...
W20260806 01:01:15.032258 8012 tablet.cc:2369] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 01:01:15.119704 8308 raft_consensus.cc:482] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 01:01:15.119889 8308 raft_consensus.cc:504] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } }
I20260806 01:01:15.120292 8308 leader_election.cc:288] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfbb4658535f4208a4d9bdb1f31c0103 (127.23.242.2:41497), 16bd3ad6217d40e19f7a6624ad9c650b (127.23.242.3:36385)
I20260806 01:01:15.123375 8299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:15.120546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:52043 (local address 127.23.242.2:41497)
0806 01:01:15.120601 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0806 01:01:15.120603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:01:15.120752 (+ 149us) server_negotiation.cc:407] Connection header received
0806 01:01:15.120826 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:01:15.120827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:01:15.120840 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:01:15.120913 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:01:15.121072 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:15.121553 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:15.122568 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:15.122988 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:15.123092 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:15.123102 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0806 01:01:15.123265 (+ 163us) server_negotiation.cc:299] Negotiation successful
0806 01:01:15.123306 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260806 01:01:15.123391 8311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:15.120491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:41497 (local address 127.23.242.1:52043)
0806 01:01:15.120660 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:15.120675 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 01:01:15.120760 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:01:15.120916 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:01:15.120917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:01:15.120919 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:01:15.121000 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:15.121001 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:15.121639 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:15.121640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:15.122468 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:15.122470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:15.123093 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:15.123094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:15.123177 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:15.123181 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 01:01:15.123246 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 01:01:15.123293 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":112,"threads_started":1}
I20260806 01:01:15.123548 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:15.120481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:36385 (local address 127.23.242.1:45877)
0806 01:01:15.120647 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:15.120659 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 01:01:15.120761 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:01:15.120932 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:01:15.120933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:01:15.120935 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:01:15.121024 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:15.121026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:15.121718 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:15.121720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:15.122552 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:15.122554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:15.123239 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:15.123241 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:15.123334 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:15.123338 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 01:01:15.123409 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 01:01:15.123454 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":81,"threads_started":1}
I20260806 01:01:15.123540 8298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:15.120559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:45877 (local address 127.23.242.3:36385)
0806 01:01:15.120609 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 01:01:15.120611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:01:15.120742 (+ 131us) server_negotiation.cc:407] Connection header received
0806 01:01:15.120837 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:01:15.120838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:01:15.120852 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:01:15.120925 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:01:15.121105 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:15.121623 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:15.122706 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:15.123132 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:15.123234 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:15.123244 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0806 01:01:15.123430 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 01:01:15.123473 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 01:01:15.123989 8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84609fa183f9435796e6335a7adb0b61" candidate_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" is_pre_election: true
I20260806 01:01:15.124188 8100 raft_consensus.cc:2454] T 84609fa183f9435796e6335a7adb0b61 P dfbb4658535f4208a4d9bdb1f31c0103 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da6c6da4a5434ab3b834d8d0eee9f57c in term 0.
I20260806 01:01:15.124315 8235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84609fa183f9435796e6335a7adb0b61" candidate_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" is_pre_election: true
I20260806 01:01:15.124485 8235 raft_consensus.cc:2454] T 84609fa183f9435796e6335a7adb0b61 P 16bd3ad6217d40e19f7a6624ad9c650b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da6c6da4a5434ab3b834d8d0eee9f57c in term 0.
I20260806 01:01:15.124483 7897 leader_election.cc:302] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da6c6da4a5434ab3b834d8d0eee9f57c, dfbb4658535f4208a4d9bdb1f31c0103; no voters:
I20260806 01:01:15.124652 8308 raft_consensus.cc:2781] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 01:01:15.124713 8308 raft_consensus.cc:482] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 01:01:15.124743 8308 raft_consensus.cc:3037] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:01:15.125527 8308 raft_consensus.cc:504] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } }
I20260806 01:01:15.125691 8308 leader_election.cc:288] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c [CANDIDATE]: Term 1 election: Requested vote from peers dfbb4658535f4208a4d9bdb1f31c0103 (127.23.242.2:41497), 16bd3ad6217d40e19f7a6624ad9c650b (127.23.242.3:36385)
I20260806 01:01:15.125875 8235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84609fa183f9435796e6335a7adb0b61" candidate_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16bd3ad6217d40e19f7a6624ad9c650b"
I20260806 01:01:15.125913 8100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84609fa183f9435796e6335a7adb0b61" candidate_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfbb4658535f4208a4d9bdb1f31c0103"
I20260806 01:01:15.125962 8235 raft_consensus.cc:3037] T 84609fa183f9435796e6335a7adb0b61 P 16bd3ad6217d40e19f7a6624ad9c650b [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:01:15.126027 8100 raft_consensus.cc:3037] T 84609fa183f9435796e6335a7adb0b61 P dfbb4658535f4208a4d9bdb1f31c0103 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:01:15.126886 8100 raft_consensus.cc:2454] T 84609fa183f9435796e6335a7adb0b61 P dfbb4658535f4208a4d9bdb1f31c0103 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6c6da4a5434ab3b834d8d0eee9f57c in term 1.
I20260806 01:01:15.127256 7897 leader_election.cc:302] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da6c6da4a5434ab3b834d8d0eee9f57c, dfbb4658535f4208a4d9bdb1f31c0103; no voters:
I20260806 01:01:15.127574 8235 raft_consensus.cc:2454] T 84609fa183f9435796e6335a7adb0b61 P 16bd3ad6217d40e19f7a6624ad9c650b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da6c6da4a5434ab3b834d8d0eee9f57c in term 1.
I20260806 01:01:15.127612 8308 raft_consensus.cc:2781] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:01:15.127800 8308 raft_consensus.cc:686] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c [term 1 LEADER]: Becoming Leader. State: Replica: da6c6da4a5434ab3b834d8d0eee9f57c, State: Running, Role: LEADER
I20260806 01:01:15.128037 8308 consensus_queue.cc:237] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } }
I20260806 01:01:15.129421 7818 catalog_manager.cc:5495] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c reported cstate change: term changed from 0 to 1, leader changed from <none> to da6c6da4a5434ab3b834d8d0eee9f57c (127.23.242.1). New cstate: current_term: 1 leader_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } health_report { overall_health: UNKNOWN } } }
I20260806 01:01:15.139822 8297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:15.136919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56518 (local address 127.23.242.1:43497)
0806 01:01:15.137044 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 01:01:15.137047 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:01:15.137058 (+ 11us) server_negotiation.cc:407] Connection header received
0806 01:01:15.137113 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:01:15.137114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:01:15.137151 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:01:15.137246 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 01:01:15.137408 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:15.137935 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:15.138914 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:15.139129 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:15.139222 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:15.139418 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 01:01:15.139535 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 01:01:15.139636 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 01:01:15.139682 (+ 46us) server_negotiation.cc:299] Negotiation successful
0806 01:01:15.139707 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 01:01:15.142208 8235 raft_consensus.cc:1261] T 84609fa183f9435796e6335a7adb0b61 P 16bd3ad6217d40e19f7a6624ad9c650b [term 1 FOLLOWER]: Refusing update from remote peer da6c6da4a5434ab3b834d8d0eee9f57c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:01:15.142204 8100 raft_consensus.cc:1261] T 84609fa183f9435796e6335a7adb0b61 P dfbb4658535f4208a4d9bdb1f31c0103 [term 1 FOLLOWER]: Refusing update from remote peer da6c6da4a5434ab3b834d8d0eee9f57c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:01:15.142655 8312 consensus_queue.cc:1035] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:01:15.142817 8308 consensus_queue.cc:1035] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c [LEADER]: Connected to new peer: Peer: permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:01:15.145541 8315 mvcc.cc:205] Tried to move back new op lower bound from 7315366195779534848 to 7315366195725561856. Current Snapshot: MvccSnapshot[applied={T|T < 7315366195779534848}]
I20260806 01:01:15.145995 8316 mvcc.cc:205] Tried to move back new op lower bound from 7315366195779534848 to 7315366195725561856. Current Snapshot: MvccSnapshot[applied={T|T < 7315366195779534848}]
I20260806 01:01:15.147397 8318 mvcc.cc:205] Tried to move back new op lower bound from 7315366195779534848 to 7315366195725561856. Current Snapshot: MvccSnapshot[applied={T|T < 7315366195779534848}]
I20260806 01:01:15.149103 7818 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 01:01:15.149233 7818 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 01:01:15.150162 7818 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5: Sending DeleteTablet for 3 replicas of tablet 84609fa183f9435796e6335a7adb0b61
I20260806 01:01:15.150727 8080 tablet_service.cc:1514] Processing DeleteTablet for tablet 84609fa183f9435796e6335a7adb0b61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:01:15 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:33306
I20260806 01:01:15.150780 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 7880
I20260806 01:01:15.150993 8215 tablet_service.cc:1514] Processing DeleteTablet for tablet 84609fa183f9435796e6335a7adb0b61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:01:15 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:49108
I20260806 01:01:15.151648 8325 tablet_replica.cc:330] T 84609fa183f9435796e6335a7adb0b61 P dfbb4658535f4208a4d9bdb1f31c0103: stopping tablet replica
I20260806 01:01:15.151787 8325 raft_consensus.cc:2229] T 84609fa183f9435796e6335a7adb0b61 P dfbb4658535f4208a4d9bdb1f31c0103 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 01:01:15.151810 8326 tablet_replica.cc:330] T 84609fa183f9435796e6335a7adb0b61 P 16bd3ad6217d40e19f7a6624ad9c650b: stopping tablet replica
I20260806 01:01:15.151919 8325 raft_consensus.cc:2258] T 84609fa183f9435796e6335a7adb0b61 P dfbb4658535f4208a4d9bdb1f31c0103 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 01:01:15.152400 8326 raft_consensus.cc:2229] T 84609fa183f9435796e6335a7adb0b61 P 16bd3ad6217d40e19f7a6624ad9c650b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 01:01:15.152536 8326 raft_consensus.cc:2258] T 84609fa183f9435796e6335a7adb0b61 P 16bd3ad6217d40e19f7a6624ad9c650b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 01:01:15.153043 8325 ts_tablet_manager.cc:1905] T 84609fa183f9435796e6335a7adb0b61 P dfbb4658535f4208a4d9bdb1f31c0103: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 01:01:15.153405 8326 ts_tablet_manager.cc:1905] T 84609fa183f9435796e6335a7adb0b61 P 16bd3ad6217d40e19f7a6624ad9c650b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 01:01:15.155608 8325 ts_tablet_manager.cc:1918] T 84609fa183f9435796e6335a7adb0b61 P dfbb4658535f4208a4d9bdb1f31c0103: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 01:01:15.155723 8325 log.cc:1196] T 84609fa183f9435796e6335a7adb0b61 P dfbb4658535f4208a4d9bdb1f31c0103: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/wal/wals/84609fa183f9435796e6335a7adb0b61
I20260806 01:01:15.155915 8326 ts_tablet_manager.cc:1918] T 84609fa183f9435796e6335a7adb0b61 P 16bd3ad6217d40e19f7a6624ad9c650b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 01:01:15.156065 8326 log.cc:1196] T 84609fa183f9435796e6335a7adb0b61 P 16bd3ad6217d40e19f7a6624ad9c650b: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/wal/wals/84609fa183f9435796e6335a7adb0b61
I20260806 01:01:15.156090 8325 ts_tablet_manager.cc:1939] T 84609fa183f9435796e6335a7adb0b61 P dfbb4658535f4208a4d9bdb1f31c0103: Deleting consensus metadata
I20260806 01:01:15.156476 8326 ts_tablet_manager.cc:1939] T 84609fa183f9435796e6335a7adb0b61 P 16bd3ad6217d40e19f7a6624ad9c650b: Deleting consensus metadata
I20260806 01:01:15.156607 7803 catalog_manager.cc:4841] TS dfbb4658535f4208a4d9bdb1f31c0103 (127.23.242.2:41497): tablet 84609fa183f9435796e6335a7adb0b61 (table test-table [id=c6b71628fcc14282b24d0a831aee6ce6]) successfully deleted
I20260806 01:01:15.157032 7804 catalog_manager.cc:4841] TS 16bd3ad6217d40e19f7a6624ad9c650b (127.23.242.3:36385): tablet 84609fa183f9435796e6335a7adb0b61 (table test-table [id=c6b71628fcc14282b24d0a831aee6ce6]) successfully deleted
W20260806 01:01:15.161243 7805 connection.cc:535] client connection to 127.23.242.1:43497 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260806 01:01:15.161821 7805 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 01:01:15.162231 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 8016
I20260806 01:01:15.163395 8294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:15.162435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:43497 (local address 127.0.0.1:56520)
0806 01:01:15.163300 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:15.163348 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.1:43497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":839}
W20260806 01:01:15.163836 7805 catalog_manager.cc:4584] TS da6c6da4a5434ab3b834d8d0eee9f57c (127.23.242.1:43497): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 84609fa183f9435796e6335a7adb0b61: Network error: Client connection negotiation failed: client connection to 127.23.242.1:43497: connect: Connection refused (error 111)
I20260806 01:01:15.171659 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 8152
I20260806 01:01:15.179538 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 7785
I20260806 01:01:35.187952 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:40935
--webserver_interface=127.23.242.62
--webserver_port=35029
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:40935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:01:35.300463 8328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:01:35.300750 8328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:01:35.300781 8328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:01:35.302146 8328 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 01:01:35.302198 8328 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 01:01:35.302212 8328 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 01:01:35.302224 8328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:01:35.302238 8328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:01:35.302253 8328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:01:35.302265 8328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:01:35.302278 8328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:01:35.304169 8328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:40935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:40935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=35029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:01:35.304494 8328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:01:35.304812 8328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:01:35.315896 8334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:01:35.316105 8328 server_base.cc:1034] running on GCE node
W20260806 01:01:35.316412 8337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:01:35.316527 8335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:01:35.316833 8328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:01:35.317204 8328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:01:35.318413 8328 hybrid_clock.cc:648] HybridClock initialized: now 1785978095318388 us; error 36 us; skew 500 ppm
Aug 06 01:01:35 dist-test-slave-dhph krb5kdc[7772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978095, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:01:35.328624 8328 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 01:01:35.329197 8328 webserver.cc:460] Webserver started at http://127.23.242.62:35029/ using document root <none> and password file <none>
I20260806 01:01:35.329489 8328 fs_manager.cc:359] Metadata directory not provided
I20260806 01:01:35.329555 8328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:01:35.331452 8328 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 01:01:35.332647 8344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:01:35.332942 8328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:01:35.333077 8328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "9e2f2c35989b42cd9f890a5776564bc5"
format_stamp: "Formatted at 2026-08-06 01:01:14 on dist-test-slave-dhph"
server_key: "9b3a0bba63de6a206fb1c8e231cbdf4b"
server_key_iv: "3139f1fd11aaefb7e7ed956a0bf4327b"
server_key_version: "encryptionkey@0"
I20260806 01:01:35.333573 8328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:01:35.361655 8328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:01:35.365963 8328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:01:35.366365 8328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:01:35.372399 8328 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:40935
I20260806 01:01:35.372447 8398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:40935 every 8 connection(s)
I20260806 01:01:35.372941 8328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:01:35.374809 8399 sys_catalog.cc:263] Verifying existing consensus state
I20260806 01:01:35.375548 8399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5: Bootstrap starting.
I20260806 01:01:35.377014 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 8328
I20260806 01:01:35.377382 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:43497
--local_ip_for_outbound_sockets=127.23.242.1
--tserver_master_addrs=127.23.242.62:40935
--webserver_port=43115
--webserver_interface=127.23.242.1
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 01:01:35.386966 8399 log.cc:824] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5: Log is configured to *not* fsync() on all Append() calls
I20260806 01:01:35.389667 8399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 01:01:35.390057 8399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5: Bootstrap complete.
I20260806 01:01:35.392711 8399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2f2c35989b42cd9f890a5776564bc5" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 40935 } }
I20260806 01:01:35.392988 8399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e2f2c35989b42cd9f890a5776564bc5, State: Initialized, Role: FOLLOWER
I20260806 01:01:35.393214 8399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2f2c35989b42cd9f890a5776564bc5" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 40935 } }
I20260806 01:01:35.393416 8399 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:01:35.393493 8399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:01:35.393548 8399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 01:01:35.394799 8399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2f2c35989b42cd9f890a5776564bc5" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 40935 } }
I20260806 01:01:35.394924 8399 leader_election.cc:302] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e2f2c35989b42cd9f890a5776564bc5; no voters:
I20260806 01:01:35.395167 8399 leader_election.cc:288] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 01:01:35.395219 8403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 01:01:35.395359 8403 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [term 2 LEADER]: Becoming Leader. State: Replica: 9e2f2c35989b42cd9f890a5776564bc5, State: Running, Role: LEADER
I20260806 01:01:35.395455 8403 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2f2c35989b42cd9f890a5776564bc5" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 40935 } }
I20260806 01:01:35.395462 8399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:01:35.395787 8405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e2f2c35989b42cd9f890a5776564bc5. Latest consensus state: current_term: 2 leader_uuid: "9e2f2c35989b42cd9f890a5776564bc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2f2c35989b42cd9f890a5776564bc5" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 40935 } } }
I20260806 01:01:35.395884 8405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [sys.catalog]: This master's current role is: LEADER
I20260806 01:01:35.396762 8412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:01:35.397426 8404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9e2f2c35989b42cd9f890a5776564bc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2f2c35989b42cd9f890a5776564bc5" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 40935 } } }
I20260806 01:01:35.397526 8404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [sys.catalog]: This master's current role is: LEADER
I20260806 01:01:35.399308 8412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:01:35.399452 8412 catalog_manager.cc:1259] Loaded cluster ID: 3128c701d3a842a3b698fe3eccd0f8db
I20260806 01:01:35.399482 8412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:01:35.400931 8412 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:01:35.401155 8412 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5: Loaded TSK: 0
I20260806 01:01:35.413172 8412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5: Generated new TSK 1
I20260806 01:01:35.413314 8412 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 01:01:35.480576 8401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:01:35.480821 8401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:01:35.480850 8401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:01:35.482280 8401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:01:35.482343 8401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:01:35.482376 8401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 01:01:35.484063 8401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:43497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=43115
--tserver_master_addrs=127.23.242.62:40935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:01:35.484377 8401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:01:35.484647 8401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:01:35.491416 8428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:01:35.491430 8427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:01:35.491430 8430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:01:35.491842 8401 server_base.cc:1034] running on GCE node
I20260806 01:01:35.492110 8401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:01:35.492463 8401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:01:35.493674 8401 hybrid_clock.cc:648] HybridClock initialized: now 1785978095493644 us; error 50 us; skew 500 ppm
Aug 06 01:01:35 dist-test-slave-dhph krb5kdc[7772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978095, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:01:35.496953 8401 init.cc:377] Logged in from keytab as kudu/127.23.242.1@KRBTEST.COM (short username kudu)
I20260806 01:01:35.497515 8401 webserver.cc:460] Webserver started at http://127.23.242.1:43115/ using document root <none> and password file <none>
I20260806 01:01:35.497793 8401 fs_manager.cc:359] Metadata directory not provided
I20260806 01:01:35.497900 8401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:01:35.499593 8401 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:01:35.500469 8437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:01:35.500756 8401 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:01:35.500891 8401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "da6c6da4a5434ab3b834d8d0eee9f57c"
format_stamp: "Formatted at 2026-08-06 01:01:14 on dist-test-slave-dhph"
server_key: "1fd76abebc85df107bb8d0b8ed92a9d2"
server_key_iv: "56379cf7796a0b50a345427f1b510d64"
server_key_version: "encryptionkey@0"
I20260806 01:01:35.501355 8401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:01:35.512542 8401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:01:35.515923 8401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:01:35.516155 8401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:01:35.516450 8401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:01:35.517096 8444 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 01:01:35.519266 8401 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 01:01:35.519335 8401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260806 01:01:35.519366 8401 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 01:01:35.520121 8401 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 01:01:35.520186 8401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260806 01:01:35.520219 8444 tablet_bootstrap.cc:492] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c: Bootstrap starting.
I20260806 01:01:35.529371 8401 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:43497
I20260806 01:01:35.529973 8401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 01:01:35.533171 8444 log.cc:824] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c: Log is configured to *not* fsync() on all Append() calls
I20260806 01:01:35.533620 8554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:43497 every 8 connection(s)
I20260806 01:01:35.535265 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 8401
I20260806 01:01:35.535686 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:41497
--local_ip_for_outbound_sockets=127.23.242.2
--tserver_master_addrs=127.23.242.62:40935
--webserver_port=38511
--webserver_interface=127.23.242.2
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 01:01:35.540503 8444 tablet_bootstrap.cc:492] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 01:01:35.540927 8444 tablet_bootstrap.cc:492] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c: Bootstrap complete.
I20260806 01:01:35.542095 8444 ts_tablet_manager.cc:1397] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.008s
I20260806 01:01:35.543421 8444 raft_consensus.cc:348] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } }
I20260806 01:01:35.543716 8444 raft_consensus.cc:729] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: da6c6da4a5434ab3b834d8d0eee9f57c, State: Initialized, Role: FOLLOWER
I20260806 01:01:35.543943 8444 consensus_queue.cc:260] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } }
I20260806 01:01:35.544348 8444 ts_tablet_manager.cc:1428] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c: Time spent starting tablet: real 0.002s user 0.006s sys 0.000s
Aug 06 01:01:35 dist-test-slave-dhph krb5kdc[7772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978095, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:01:35.555215 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:35.531057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.1:56211)
0806 01:01:35.533909 (+ 2852us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:35.533958 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 01:01:35.534526 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:01:35.539206 (+ 4680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:01:35.539211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:01:35.539638 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:01:35.539931 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:35.539940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.544122 (+ 4182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.544123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:35.545343 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:35.545347 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.546250 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.546251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:35.546400 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:35.547072 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:01:35.547082 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:01:35.550153 (+ 3071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:01:35.553113 (+ 2960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:01:35.553124 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:01:35.553127 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:01:35.553483 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:01:35.553815 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:01:35.553816 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:01:35.553816 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:01:35.553946 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:01:35.554449 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:01:35.554456 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:01:35.554640 (+ 184us) client_negotiation.cc:769] Sending connection context
0806 01:01:35.554769 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 01:01:35.554968 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2799,"thread_start_us":127,"threads_started":1}
I20260806 01:01:35.556107 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:35.531218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:56211 (local address 127.23.242.62:40935)
0806 01:01:35.531409 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 01:01:35.531418 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 01:01:35.534211 (+ 2793us) server_negotiation.cc:407] Connection header received
0806 01:01:35.538782 (+ 4571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:01:35.538792 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:01:35.538872 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:01:35.540110 (+ 1238us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:01:35.542384 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.543928 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.545456 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.546040 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.546148 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:35.550335 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:01:35.550341 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:01:35.550341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:01:35.550357 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:01:35.552981 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:01:35.553597 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:01:35.553602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:01:35.553602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:01:35.553724 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:01:35.554017 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:01:35.554018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:01:35.554018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:01:35.554336 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:01:35.554409 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 01:01:35.555424 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0806 01:01:35.555599 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":80,"threads_started":1}
I20260806 01:01:35.559023 8555 heartbeater.cc:347] Connected to a master server at 127.23.242.62:40935
I20260806 01:01:35.559166 8555 heartbeater.cc:464] Registering TS with master...
I20260806 01:01:35.559960 8555 heartbeater.cc:511] Master 127.23.242.62:40935 requested a full tablet report, sending...
I20260806 01:01:35.561477 8361 ts_manager.cc:194] Registered new tserver with Master: da6c6da4a5434ab3b834d8d0eee9f57c (127.23.242.1:43497)
I20260806 01:01:35.562944 8361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.1@KRBTEST.COM'} at 127.23.242.1:56211
I20260806 01:01:35.564167 8555 heartbeater.cc:503] Master 127.23.242.62:40935 was elected leader, sending a full tablet report...
I20260806 01:01:35.568445 8562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:35.562284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:43497 (local address 127.0.0.1:42840)
0806 01:01:35.563383 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:35.563424 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 01:01:35.563504 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:01:35.564151 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:01:35.564155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:01:35.564158 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:01:35.564248 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:35.564250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.565790 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.565791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:35.567170 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:35.567175 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.568122 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.568123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:35.568213 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:35.568228 (+ 15us) client_negotiation.cc:769] Sending connection context
0806 01:01:35.568300 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 01:01:35.568353 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1066,"thread_start_us":98,"threads_started":1}
I20260806 01:01:35.568465 8563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:35.562639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42840 (local address 127.23.242.1:43497)
0806 01:01:35.563678 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0806 01:01:35.563684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 01:01:35.563697 (+ 13us) server_negotiation.cc:407] Connection header received
0806 01:01:35.563752 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:01:35.563754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:01:35.564019 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:01:35.564331 (+ 312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:01:35.564385 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.565706 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.567273 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.568025 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.568112 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:35.568182 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 01:01:35.568337 (+ 155us) server_negotiation.cc:299] Negotiation successful
0806 01:01:35.568383 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1000,"thread_start_us":107,"threads_started":1}
I20260806 01:01:35.568850 8469 tablet_service.cc:1514] Processing DeleteTablet for tablet 84609fa183f9435796e6335a7adb0b61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:01:15 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:42840
I20260806 01:01:35.569609 8564 tablet_replica.cc:330] stopping tablet replica
I20260806 01:01:35.569839 8564 raft_consensus.cc:2229] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 01:01:35.569962 8564 raft_consensus.cc:2258] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 01:01:35.570420 8564 ts_tablet_manager.cc:1905] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 01:01:35.574455 8564 ts_tablet_manager.cc:1918] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 01:01:35.574633 8564 log.cc:1196] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/wal/wals/84609fa183f9435796e6335a7adb0b61
I20260806 01:01:35.575003 8564 ts_tablet_manager.cc:1939] T 84609fa183f9435796e6335a7adb0b61 P da6c6da4a5434ab3b834d8d0eee9f57c: Deleting consensus metadata
I20260806 01:01:35.575711 8348 catalog_manager.cc:4841] TS da6c6da4a5434ab3b834d8d0eee9f57c (127.23.242.1:43497): tablet 84609fa183f9435796e6335a7adb0b61 (table test-table [id=c6b71628fcc14282b24d0a831aee6ce6]) successfully deleted
W20260806 01:01:35.646555 8560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:01:35.647003 8560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:01:35.647038 8560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:01:35.648551 8560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:01:35.648614 8560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:01:35.648651 8560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 01:01:35.650441 8560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:41497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=38511
--tserver_master_addrs=127.23.242.62:40935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:01:35.650802 8560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:01:35.651093 8560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:01:35.657542 8569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:01:35.657698 8572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:01:35.657815 8560 server_base.cc:1034] running on GCE node
W20260806 01:01:35.657699 8570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:01:35.658233 8560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:01:35.658471 8560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:01:35.659632 8560 hybrid_clock.cc:648] HybridClock initialized: now 1785978095659624 us; error 52 us; skew 500 ppm
Aug 06 01:01:35 dist-test-slave-dhph krb5kdc[7772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978095, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:01:35.663233 8560 init.cc:377] Logged in from keytab as kudu/127.23.242.2@KRBTEST.COM (short username kudu)
I20260806 01:01:35.663869 8560 webserver.cc:460] Webserver started at http://127.23.242.2:38511/ using document root <none> and password file <none>
I20260806 01:01:35.664194 8560 fs_manager.cc:359] Metadata directory not provided
I20260806 01:01:35.664273 8560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:01:35.666726 8560 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 01:01:35.668044 8579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:01:35.668447 8560 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:01:35.668597 8560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "dfbb4658535f4208a4d9bdb1f31c0103"
format_stamp: "Formatted at 2026-08-06 01:01:14 on dist-test-slave-dhph"
server_key: "67f389e1ed8bdbf335c95c1873eecbb2"
server_key_iv: "1e9f00d66187292452ce4eec974ab02a"
server_key_version: "encryptionkey@0"
I20260806 01:01:35.669030 8560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:01:35.681809 8560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:01:35.684787 8560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:01:35.684978 8560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:01:35.685254 8560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:01:35.685591 8560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:01:35.685647 8560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:01:35.685683 8560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:01:35.685715 8560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:01:35.692186 8560 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:41497
I20260806 01:01:35.692438 8695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:41497 every 8 connection(s)
I20260806 01:01:35.692672 8560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 01:01:35.696158 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 8560
I20260806 01:01:35.696619 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:36385
--local_ip_for_outbound_sockets=127.23.242.3
--tserver_master_addrs=127.23.242.62:40935
--webserver_port=43959
--webserver_interface=127.23.242.3
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 01:01:35 dist-test-slave-dhph krb5kdc[7772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978095, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:01:35.706674 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:35.693959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:42009 (local address 127.23.242.62:40935)
0806 01:01:35.694037 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0806 01:01:35.694039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:01:35.694051 (+ 12us) server_negotiation.cc:407] Connection header received
0806 01:01:35.694629 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:01:35.694630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:01:35.694645 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:01:35.694713 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:01:35.695856 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.697454 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.699335 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.699593 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.699690 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:35.702847 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:01:35.702850 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:01:35.702850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:01:35.702859 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:01:35.704844 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:01:35.705320 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:01:35.705321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:01:35.705322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:01:35.705414 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:01:35.705724 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:01:35.705725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:01:35.705725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:01:35.705856 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:01:35.705936 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 01:01:35.706421 (+ 485us) server_negotiation.cc:299] Negotiation successful
0806 01:01:35.706537 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 01:01:35.706962 8698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:35.693581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.2:42009)
0806 01:01:35.693900 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:35.693953 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 01:01:35.694480 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:01:35.694799 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:01:35.694802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:01:35.695526 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:01:35.695738 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:35.695753 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.697577 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.697579 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:35.699191 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:35.699195 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.699700 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.699701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:35.699883 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:35.700540 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:01:35.700550 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:01:35.702511 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:01:35.704956 (+ 2445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:01:35.704965 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:01:35.704967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:01:35.705233 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:01:35.705496 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:01:35.705497 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:01:35.705497 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:01:35.705649 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:01:35.705926 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:01:35.705934 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:01:35.706197 (+ 263us) client_negotiation.cc:769] Sending connection context
0806 01:01:35.706390 (+ 193us) client_negotiation.cc:240] Negotiation successful
0806 01:01:35.706627 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":192,"threads_started":1}
I20260806 01:01:35.707638 8696 heartbeater.cc:347] Connected to a master server at 127.23.242.62:40935
I20260806 01:01:35.707969 8696 heartbeater.cc:464] Registering TS with master...
I20260806 01:01:35.708264 8696 heartbeater.cc:511] Master 127.23.242.62:40935 requested a full tablet report, sending...
I20260806 01:01:35.708783 8361 ts_manager.cc:194] Registered new tserver with Master: dfbb4658535f4208a4d9bdb1f31c0103 (127.23.242.2:41497)
I20260806 01:01:35.709192 8361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.2@KRBTEST.COM'} at 127.23.242.2:42009
W20260806 01:01:35.801997 8699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:01:35.802264 8699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:01:35.802295 8699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:01:35.803808 8699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:01:35.803869 8699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:01:35.803901 8699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 01:01:35.805708 8699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:36385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=43959
--tserver_master_addrs=127.23.242.62:40935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:01:35.806003 8699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:01:35.806454 8699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:01:35.813493 8705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:01:35.813510 8708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:01:35.813493 8706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:01:35.814265 8699 server_base.cc:1034] running on GCE node
I20260806 01:01:35.814524 8699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:01:35.814880 8699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:01:35.816093 8699 hybrid_clock.cc:648] HybridClock initialized: now 1785978095816088 us; error 43 us; skew 500 ppm
Aug 06 01:01:35 dist-test-slave-dhph krb5kdc[7772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978095, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:01:35.819648 8699 init.cc:377] Logged in from keytab as kudu/127.23.242.3@KRBTEST.COM (short username kudu)
I20260806 01:01:35.820204 8699 webserver.cc:460] Webserver started at http://127.23.242.3:43959/ using document root <none> and password file <none>
I20260806 01:01:35.820480 8699 fs_manager.cc:359] Metadata directory not provided
I20260806 01:01:35.820545 8699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:01:35.822134 8699 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:01:35.823410 8715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:01:35.823719 8699 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 01:01:35.823858 8699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "16bd3ad6217d40e19f7a6624ad9c650b"
format_stamp: "Formatted at 2026-08-06 01:01:14 on dist-test-slave-dhph"
server_key: "cc6ba1f343637f722f44cccdb65dcc64"
server_key_iv: "3f5803827a1f305886458c8f4faaf384"
server_key_version: "encryptionkey@0"
I20260806 01:01:35.824306 8699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:01:35.843379 8699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:01:35.845935 8699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:01:35.846071 8699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:01:35.846359 8699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:01:35.847730 8699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:01:35.847822 8699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:01:35.847859 8699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:01:35.847873 8699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:01:35.856617 8699 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:36385
I20260806 01:01:35.856722 8829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:36385 every 8 connection(s)
I20260806 01:01:35.857439 8699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
I20260806 01:01:35.864167 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 8699
Aug 06 01:01:35 dist-test-slave-dhph krb5kdc[7772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978095, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:01:35.873754 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:35.858978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:54383 (local address 127.23.242.62:40935)
0806 01:01:35.859097 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 01:01:35.859099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:01:35.859394 (+ 295us) server_negotiation.cc:407] Connection header received
0806 01:01:35.859987 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:01:35.859994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:01:35.860012 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:01:35.860065 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:01:35.862539 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.863164 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.864941 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.865293 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.865354 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:35.869545 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:01:35.869547 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:01:35.869548 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:01:35.869557 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:01:35.871722 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:01:35.872345 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:01:35.872346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:01:35.872347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:01:35.872425 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:01:35.872929 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:01:35.872930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:01:35.872931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:01:35.873072 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:01:35.873186 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 01:01:35.873548 (+ 362us) server_negotiation.cc:299] Negotiation successful
0806 01:01:35.873642 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 01:01:35.874213 8832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:35.858601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.3:54383)
0806 01:01:35.859192 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:35.859236 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 01:01:35.859774 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:01:35.860203 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:01:35.860207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:01:35.861666 (+ 1459us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:01:35.862378 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:35.862388 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.863328 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.863329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:35.864800 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:35.864803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.865564 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.865565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:35.865782 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:35.866987 (+ 1205us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:01:35.866996 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:01:35.869404 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:01:35.871818 (+ 2414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:01:35.871833 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:01:35.871836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:01:35.872193 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:01:35.872507 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:01:35.872508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:01:35.872509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:01:35.872669 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:01:35.873182 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:01:35.873192 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:01:35.873384 (+ 192us) client_negotiation.cc:769] Sending connection context
0806 01:01:35.873526 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 01:01:35.873736 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":354,"threads_started":1}
I20260806 01:01:35.874841 8830 heartbeater.cc:347] Connected to a master server at 127.23.242.62:40935
I20260806 01:01:35.874986 8830 heartbeater.cc:464] Registering TS with master...
I20260806 01:01:35.875058 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:35.864746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52722 (local address 127.23.242.62:40935)
0806 01:01:35.864898 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 01:01:35.864901 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:01:35.865016 (+ 115us) server_negotiation.cc:407] Connection header received
0806 01:01:35.865044 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:01:35.865045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:01:35.865058 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:01:35.865168 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:01:35.866427 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.867083 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.868150 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.868492 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.868767 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:35.870380 (+ 1613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:01:35.870381 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:01:35.870382 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:01:35.870390 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:01:35.872777 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:01:35.873278 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:01:35.873279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:01:35.873280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:01:35.873337 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:01:35.873656 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:01:35.873657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:01:35.873658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:01:35.873837 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:01:35.874276 (+ 439us) server_negotiation.cc:1035] Waiting for connection context
0806 01:01:35.874833 (+ 557us) server_negotiation.cc:299] Negotiation successful
0806 01:01:35.874942 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
I20260806 01:01:35.875259 8830 heartbeater.cc:511] Master 127.23.242.62:40935 requested a full tablet report, sending...
I20260806 01:01:35.876569 8360 ts_manager.cc:194] Registered new tserver with Master: 16bd3ad6217d40e19f7a6624ad9c650b (127.23.242.3:36385)
I20260806 01:01:35.877295 8360 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.3@KRBTEST.COM'} at 127.23.242.3:54383
I20260806 01:01:35.888775 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 01:01:35.892416 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:35.889476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52736 (local address 127.23.242.62:40935)
0806 01:01:35.889554 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0806 01:01:35.889556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:01:35.889648 (+ 92us) server_negotiation.cc:407] Connection header received
0806 01:01:35.889704 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:01:35.889705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:01:35.889747 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:01:35.889828 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 01:01:35.890012 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.890645 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.891500 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.891754 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.891841 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:35.892038 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 01:01:35.892104 (+ 66us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0806 01:01:35.892324 (+ 220us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52736: token expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 01:01:35.892511 8833 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52736: token expired
W20260806 01:01:35.892876 8286 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.23.242.62:40935)
I20260806 01:01:35.901963 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:35.893318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52738 (local address 127.23.242.62:40935)
0806 01:01:35.893408 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0806 01:01:35.893411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:01:35.893423 (+ 12us) server_negotiation.cc:407] Connection header received
0806 01:01:35.893478 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:01:35.893479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:01:35.893493 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:01:35.893564 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:01:35.894170 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.894764 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.895725 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.896019 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.896115 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:35.897708 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:01:35.897711 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:01:35.897711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:01:35.897722 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:01:35.900335 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:01:35.900815 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:01:35.900816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:01:35.900816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:01:35.900879 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:01:35.901239 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:01:35.901240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:01:35.901240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:01:35.901440 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:01:35.901542 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 01:01:35.901791 (+ 249us) server_negotiation.cc:299] Negotiation successful
0806 01:01:35.901884 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 01:01:35.904450 8360 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 01:01:35.907694 8360 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 01:01:35.910981 8469 tablet_service.cc:1467] Processing CreateTablet for tablet af421777b0be4879a470c36d8f3d4e0f (DEFAULT_TABLE table=test-table [id=df482e08f6b8433198c3e4701869abae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:01:35.911260 8469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af421777b0be4879a470c36d8f3d4e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:01:35.912693 8444 tablet_bootstrap.cc:492] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c: Bootstrap starting.
I20260806 01:01:35.913134 8444 tablet_bootstrap.cc:654] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c: Neither blocks nor log segments found. Creating new log.
I20260806 01:01:35.913765 8444 tablet_bootstrap.cc:492] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c: No bootstrap required, opened a new log
I20260806 01:01:35.913838 8444 ts_tablet_manager.cc:1397] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 01:01:35.913964 8444 raft_consensus.cc:348] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } }
I20260806 01:01:35.914067 8444 raft_consensus.cc:374] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:01:35.914129 8444 raft_consensus.cc:729] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da6c6da4a5434ab3b834d8d0eee9f57c, State: Initialized, Role: FOLLOWER
I20260806 01:01:35.914175 8444 consensus_queue.cc:260] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } }
I20260806 01:01:35.914286 8444 ts_tablet_manager.cc:1428] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260806 01:01:35.916085 8836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:35.910566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56738 (local address 127.23.242.2:41497)
0806 01:01:35.911219 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0806 01:01:35.911224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 01:01:35.911238 (+ 14us) server_negotiation.cc:407] Connection header received
0806 01:01:35.911287 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:01:35.911289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:01:35.911306 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:01:35.911638 (+ 332us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:01:35.911731 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.912502 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.914045 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.915359 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.915440 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:35.915501 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 01:01:35.915976 (+ 475us) server_negotiation.cc:299] Negotiation successful
0806 01:01:35.916018 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":219,"threads_started":1}
I20260806 01:01:35.916167 8562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:35.910433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:41497 (local address 127.0.0.1:56738)
0806 01:01:35.910488 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:35.910499 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 01:01:35.910544 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:01:35.911459 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:01:35.911460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:01:35.911463 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:01:35.911526 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:35.911529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.912600 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.912601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:35.913946 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:35.913950 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.915563 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.915564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:35.915829 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:35.915834 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 01:01:35.915952 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 01:01:35.916013 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260806 01:01:35.917258 8837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:35.910626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:36385 (local address 127.0.0.1:48888)
0806 01:01:35.912853 (+ 2227us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:35.912867 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 01:01:35.912945 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:01:35.913160 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:01:35.913162 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:01:35.913164 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:01:35.913220 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:35.913222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.914834 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.914835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:35.915803 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:35.915806 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.917008 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.917010 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:35.917071 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:35.917075 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 01:01:35.917148 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 01:01:35.917186 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2206,"thread_start_us":151,"threads_started":1}
I20260806 01:01:35.917130 8629 tablet_service.cc:1467] Processing CreateTablet for tablet af421777b0be4879a470c36d8f3d4e0f (DEFAULT_TABLE table=test-table [id=df482e08f6b8433198c3e4701869abae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:01:35.917316 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:35.911266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48888 (local address 127.23.242.3:36385)
0806 01:01:35.911497 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 01:01:35.911502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 01:01:35.912964 (+ 1462us) server_negotiation.cc:407] Connection header received
0806 01:01:35.913024 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:01:35.913027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:01:35.913055 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:01:35.913146 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:01:35.913316 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.914738 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.915919 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.916836 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.916919 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:35.916972 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 01:01:35.917168 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 01:01:35.917221 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":139,"threads_started":1}
I20260806 01:01:35.917722 8629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af421777b0be4879a470c36d8f3d4e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:01:35.920836 8764 tablet_service.cc:1467] Processing CreateTablet for tablet af421777b0be4879a470c36d8f3d4e0f (DEFAULT_TABLE table=test-table [id=df482e08f6b8433198c3e4701869abae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:01:35.921676 8764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af421777b0be4879a470c36d8f3d4e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:01:35.921923 8840 tablet_bootstrap.cc:492] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103: Bootstrap starting.
I20260806 01:01:35.923085 8840 tablet_bootstrap.cc:654] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103: Neither blocks nor log segments found. Creating new log.
I20260806 01:01:35.923760 8840 log.cc:824] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103: Log is configured to *not* fsync() on all Append() calls
I20260806 01:01:35.924657 8840 tablet_bootstrap.cc:492] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103: No bootstrap required, opened a new log
I20260806 01:01:35.924747 8840 ts_tablet_manager.cc:1397] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260806 01:01:35.926489 8842 tablet_bootstrap.cc:492] T af421777b0be4879a470c36d8f3d4e0f P 16bd3ad6217d40e19f7a6624ad9c650b: Bootstrap starting.
I20260806 01:01:35.927050 8840 raft_consensus.cc:348] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } }
I20260806 01:01:35.927271 8840 raft_consensus.cc:374] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:01:35.927326 8840 raft_consensus.cc:729] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfbb4658535f4208a4d9bdb1f31c0103, State: Initialized, Role: FOLLOWER
I20260806 01:01:35.927455 8840 consensus_queue.cc:260] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } }
I20260806 01:01:35.927507 8842 tablet_bootstrap.cc:654] T af421777b0be4879a470c36d8f3d4e0f P 16bd3ad6217d40e19f7a6624ad9c650b: Neither blocks nor log segments found. Creating new log.
I20260806 01:01:35.927868 8696 heartbeater.cc:503] Master 127.23.242.62:40935 was elected leader, sending a full tablet report...
I20260806 01:01:35.928172 8840 ts_tablet_manager.cc:1428] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260806 01:01:35.928256 8842 log.cc:824] T af421777b0be4879a470c36d8f3d4e0f P 16bd3ad6217d40e19f7a6624ad9c650b: Log is configured to *not* fsync() on all Append() calls
I20260806 01:01:35.929206 8842 tablet_bootstrap.cc:492] T af421777b0be4879a470c36d8f3d4e0f P 16bd3ad6217d40e19f7a6624ad9c650b: No bootstrap required, opened a new log
I20260806 01:01:35.929315 8842 ts_tablet_manager.cc:1397] T af421777b0be4879a470c36d8f3d4e0f P 16bd3ad6217d40e19f7a6624ad9c650b: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260806 01:01:35.930898 8842 raft_consensus.cc:348] T af421777b0be4879a470c36d8f3d4e0f P 16bd3ad6217d40e19f7a6624ad9c650b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } }
I20260806 01:01:35.931054 8842 raft_consensus.cc:374] T af421777b0be4879a470c36d8f3d4e0f P 16bd3ad6217d40e19f7a6624ad9c650b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:01:35.931078 8842 raft_consensus.cc:729] T af421777b0be4879a470c36d8f3d4e0f P 16bd3ad6217d40e19f7a6624ad9c650b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16bd3ad6217d40e19f7a6624ad9c650b, State: Initialized, Role: FOLLOWER
I20260806 01:01:35.931195 8842 consensus_queue.cc:260] T af421777b0be4879a470c36d8f3d4e0f P 16bd3ad6217d40e19f7a6624ad9c650b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } }
I20260806 01:01:35.932520 8842 ts_tablet_manager.cc:1428] T af421777b0be4879a470c36d8f3d4e0f P 16bd3ad6217d40e19f7a6624ad9c650b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 01:01:35.932524 8830 heartbeater.cc:503] Master 127.23.242.62:40935 was elected leader, sending a full tablet report...
W20260806 01:01:35.943430 8697 tablet.cc:2369] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 01:01:35.957882 8843 raft_consensus.cc:482] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 01:01:35.958065 8843 raft_consensus.cc:504] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } }
I20260806 01:01:35.958477 8843 leader_election.cc:288] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16bd3ad6217d40e19f7a6624ad9c650b (127.23.242.3:36385), da6c6da4a5434ab3b834d8d0eee9f57c (127.23.242.1:43497)
I20260806 01:01:35.961766 8698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:35.958753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:43497 (local address 127.23.242.2:45355)
0806 01:01:35.958843 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:35.958860 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 01:01:35.958928 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:01:35.959109 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:01:35.959111 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:01:35.959113 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:01:35.959251 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:35.959254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.959917 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.959918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:35.960837 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:35.960840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.961493 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.961494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:35.961602 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:35.961606 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 01:01:35.961668 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 01:01:35.961704 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 01:01:35.961872 8563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:35.958870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:45355 (local address 127.23.242.1:43497)
0806 01:01:35.958918 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 01:01:35.958920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:01:35.958932 (+ 12us) server_negotiation.cc:407] Connection header received
0806 01:01:35.959015 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:01:35.959016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:01:35.959028 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:01:35.959088 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:01:35.959341 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.959831 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.961036 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.961413 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.961470 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:35.961477 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0806 01:01:35.961707 (+ 230us) server_negotiation.cc:299] Negotiation successful
0806 01:01:35.961758 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 01:01:35.962023 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:35.958927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:60747 (local address 127.23.242.3:36385)
0806 01:01:35.958978 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0806 01:01:35.958980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:01:35.959092 (+ 112us) server_negotiation.cc:407] Connection header received
0806 01:01:35.959125 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:01:35.959126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:01:35.959136 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:01:35.959203 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:01:35.959377 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.959967 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.961045 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.961493 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.961574 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:35.961583 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 01:01:35.961897 (+ 314us) server_negotiation.cc:299] Negotiation successful
0806 01:01:35.961948 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260806 01:01:35.962248 8846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:35.958762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:36385 (local address 127.23.242.2:60747)
0806 01:01:35.958996 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:35.959006 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 01:01:35.959078 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:01:35.959219 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:01:35.959219 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:01:35.959221 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:01:35.959280 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:35.959282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.960082 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.960083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:35.960840 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:35.960842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.961631 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.961632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:35.961782 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:35.961788 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 01:01:35.962110 (+ 322us) client_negotiation.cc:240] Negotiation successful
0806 01:01:35.962162 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":56,"thread_start_us":127,"threads_started":1}
I20260806 01:01:35.962518 8507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af421777b0be4879a470c36d8f3d4e0f" candidate_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" is_pre_election: true
I20260806 01:01:35.962723 8507 raft_consensus.cc:2454] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfbb4658535f4208a4d9bdb1f31c0103 in term 0.
I20260806 01:01:35.962667 8784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af421777b0be4879a470c36d8f3d4e0f" candidate_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" is_pre_election: true
I20260806 01:01:35.962859 8784 raft_consensus.cc:2454] T af421777b0be4879a470c36d8f3d4e0f P 16bd3ad6217d40e19f7a6624ad9c650b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfbb4658535f4208a4d9bdb1f31c0103 in term 0.
I20260806 01:01:35.963030 8583 leader_election.cc:302] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da6c6da4a5434ab3b834d8d0eee9f57c, dfbb4658535f4208a4d9bdb1f31c0103; no voters:
I20260806 01:01:35.963213 8843 raft_consensus.cc:2781] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 01:01:35.963259 8843 raft_consensus.cc:482] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 01:01:35.963313 8843 raft_consensus.cc:3037] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:01:35.964426 8843 raft_consensus.cc:504] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } }
I20260806 01:01:35.964717 8843 leader_election.cc:288] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103 [CANDIDATE]: Term 1 election: Requested vote from peers 16bd3ad6217d40e19f7a6624ad9c650b (127.23.242.3:36385), da6c6da4a5434ab3b834d8d0eee9f57c (127.23.242.1:43497)
I20260806 01:01:35.965018 8507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af421777b0be4879a470c36d8f3d4e0f" candidate_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c"
I20260806 01:01:35.965029 8784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af421777b0be4879a470c36d8f3d4e0f" candidate_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16bd3ad6217d40e19f7a6624ad9c650b"
I20260806 01:01:35.965138 8507 raft_consensus.cc:3037] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:01:35.965165 8784 raft_consensus.cc:3037] T af421777b0be4879a470c36d8f3d4e0f P 16bd3ad6217d40e19f7a6624ad9c650b [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:01:35.966307 8784 raft_consensus.cc:2454] T af421777b0be4879a470c36d8f3d4e0f P 16bd3ad6217d40e19f7a6624ad9c650b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfbb4658535f4208a4d9bdb1f31c0103 in term 1.
I20260806 01:01:35.966444 8507 raft_consensus.cc:2454] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfbb4658535f4208a4d9bdb1f31c0103 in term 1.
I20260806 01:01:35.966787 8582 leader_election.cc:302] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16bd3ad6217d40e19f7a6624ad9c650b, dfbb4658535f4208a4d9bdb1f31c0103; no voters:
I20260806 01:01:35.966938 8843 raft_consensus.cc:2781] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:01:35.967123 8843 raft_consensus.cc:686] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103 [term 1 LEADER]: Becoming Leader. State: Replica: dfbb4658535f4208a4d9bdb1f31c0103, State: Running, Role: LEADER
I20260806 01:01:35.967238 8843 consensus_queue.cc:237] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } }
I20260806 01:01:35.968212 8360 catalog_manager.cc:5495] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfbb4658535f4208a4d9bdb1f31c0103 (127.23.242.2). New cstate: current_term: 1 leader_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } health_report { overall_health: UNKNOWN } } }
I20260806 01:01:35.981328 8836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:35.978681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56746 (local address 127.23.242.2:41497)
0806 01:01:35.978749 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0806 01:01:35.978751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:01:35.978760 (+ 9us) server_negotiation.cc:407] Connection header received
0806 01:01:35.978795 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:01:35.978796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:01:35.978815 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:01:35.978901 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 01:01:35.979040 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.979543 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.980342 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:35.980579 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:35.980681 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:35.980900 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 01:01:35.981027 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 01:01:35.981128 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 01:01:35.981179 (+ 51us) server_negotiation.cc:299] Negotiation successful
0806 01:01:35.981203 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 01:01:35.983886 8507 raft_consensus.cc:1261] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [term 1 FOLLOWER]: Refusing update from remote peer dfbb4658535f4208a4d9bdb1f31c0103: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:01:35.984040 8784 raft_consensus.cc:1261] T af421777b0be4879a470c36d8f3d4e0f P 16bd3ad6217d40e19f7a6624ad9c650b [term 1 FOLLOWER]: Refusing update from remote peer dfbb4658535f4208a4d9bdb1f31c0103: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:01:35.984346 8847 consensus_queue.cc:1035] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:01:35.984490 8843 consensus_queue.cc:1035] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:01:35.987952 8851 mvcc.cc:205] Tried to move back new op lower bound from 7315366281146236928 to 7315366281082597376. Current Snapshot: MvccSnapshot[applied={T|T < 7315366281146236928}]
I20260806 01:01:35.988422 8852 mvcc.cc:205] Tried to move back new op lower bound from 7315366281146236928 to 7315366281082597376. Current Snapshot: MvccSnapshot[applied={T|T < 7315366281146236928}]
I20260806 01:01:35.990123 8361 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 01:01:35.990648 8361 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 01:01:35.992012 8361 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5: Sending DeleteTablet for 3 replicas of tablet af421777b0be4879a470c36d8f3d4e0f
I20260806 01:01:35.992425 8629 tablet_service.cc:1514] Processing DeleteTablet for tablet af421777b0be4879a470c36d8f3d4e0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:01:35 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:56738
I20260806 01:01:35.992544 8469 tablet_service.cc:1514] Processing DeleteTablet for tablet af421777b0be4879a470c36d8f3d4e0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:01:35 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:42840
I20260806 01:01:35.992748 8564 tablet_replica.cc:330] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c: stopping tablet replica
I20260806 01:01:35.992831 8861 tablet_replica.cc:330] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103: stopping tablet replica
I20260806 01:01:35.992848 8564 raft_consensus.cc:2229] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 01:01:35.992828 8764 tablet_service.cc:1514] Processing DeleteTablet for tablet af421777b0be4879a470c36d8f3d4e0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:01:35 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:48888
I20260806 01:01:35.992942 8564 raft_consensus.cc:2258] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 01:01:35.992997 8861 raft_consensus.cc:2229] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103 [term 1 LEADER]: Raft consensus shutting down.
I20260806 01:01:35.993119 8861 raft_consensus.cc:2258] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 01:01:35.993129 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 8401
I20260806 01:01:35.993664 8861 ts_tablet_manager.cc:1905] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 01:01:35.997879 8861 ts_tablet_manager.cc:1918] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 01:01:35.998028 8861 log.cc:1196] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/wal/wals/af421777b0be4879a470c36d8f3d4e0f
I20260806 01:01:35.998422 8861 ts_tablet_manager.cc:1939] T af421777b0be4879a470c36d8f3d4e0f P dfbb4658535f4208a4d9bdb1f31c0103: Deleting consensus metadata
I20260806 01:01:35.999039 8346 catalog_manager.cc:4841] TS dfbb4658535f4208a4d9bdb1f31c0103 (127.23.242.2:41497): tablet af421777b0be4879a470c36d8f3d4e0f (table test-table [id=df482e08f6b8433198c3e4701869abae]) successfully deleted
W20260806 01:01:36.002607 8348 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.242.1:43497 (error 108)
I20260806 01:01:36.003013 8862 tablet_replica.cc:330] T af421777b0be4879a470c36d8f3d4e0f P 16bd3ad6217d40e19f7a6624ad9c650b: stopping tablet replica
I20260806 01:01:36.003095 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 8560
I20260806 01:01:36.003374 8837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:36.003094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:43497 (local address 127.0.0.1:42848)
0806 01:01:36.003178 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:36.003216 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.1:43497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260806 01:01:36.004446 8348 catalog_manager.cc:4584] TS da6c6da4a5434ab3b834d8d0eee9f57c (127.23.242.1:43497): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet af421777b0be4879a470c36d8f3d4e0f: Network error: Client connection negotiation failed: client connection to 127.23.242.1:43497: connect: Connection refused (error 111)
I20260806 01:01:36.005908 8862 raft_consensus.cc:2229] T af421777b0be4879a470c36d8f3d4e0f P 16bd3ad6217d40e19f7a6624ad9c650b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 01:01:36.006071 8862 raft_consensus.cc:2258] T af421777b0be4879a470c36d8f3d4e0f P 16bd3ad6217d40e19f7a6624ad9c650b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 01:01:36.006704 8862 ts_tablet_manager.cc:1905] T af421777b0be4879a470c36d8f3d4e0f P 16bd3ad6217d40e19f7a6624ad9c650b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 01:01:36.008577 8862 ts_tablet_manager.cc:1918] T af421777b0be4879a470c36d8f3d4e0f P 16bd3ad6217d40e19f7a6624ad9c650b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 01:01:36.008664 8862 log.cc:1196] T af421777b0be4879a470c36d8f3d4e0f P 16bd3ad6217d40e19f7a6624ad9c650b: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/wal/wals/af421777b0be4879a470c36d8f3d4e0f
I20260806 01:01:36.008963 8862 ts_tablet_manager.cc:1939] T af421777b0be4879a470c36d8f3d4e0f P 16bd3ad6217d40e19f7a6624ad9c650b: Deleting consensus metadata
I20260806 01:01:36.009508 8347 catalog_manager.cc:4841] TS 16bd3ad6217d40e19f7a6624ad9c650b (127.23.242.3:36385): tablet af421777b0be4879a470c36d8f3d4e0f (table test-table [id=df482e08f6b8433198c3e4701869abae]) successfully deleted
I20260806 01:01:36.010778 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 8699
I20260806 01:01:36.017958 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 8328
I20260806 01:01:56.026858 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:40935
--webserver_interface=127.23.242.62
--webserver_port=35029
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:40935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:01:56.132512 8864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:01:56.132795 8864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:01:56.132823 8864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:01:56.134264 8864 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 01:01:56.134320 8864 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 01:01:56.134333 8864 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 01:01:56.134346 8864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:01:56.134361 8864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:01:56.134373 8864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:01:56.134387 8864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:01:56.134435 8864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:01:56.136237 8864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:40935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:40935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=35029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:01:56.136539 8864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:01:56.136837 8864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 01:01:56.144107 8864 server_base.cc:1034] running on GCE node
W20260806 01:01:56.144107 8869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:01:56.144230 8870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:01:56.144230 8872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:01:56.144906 8864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:01:56.145218 8864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:01:56.146499 8864 hybrid_clock.cc:648] HybridClock initialized: now 1785978116146456 us; error 61 us; skew 500 ppm
Aug 06 01:01:56 dist-test-slave-dhph krb5kdc[7772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978116, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:01:56.156520 8864 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 01:01:56.157151 8864 webserver.cc:460] Webserver started at http://127.23.242.62:35029/ using document root <none> and password file <none>
I20260806 01:01:56.157455 8864 fs_manager.cc:359] Metadata directory not provided
I20260806 01:01:56.157545 8864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:01:56.159579 8864 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 01:01:56.160787 8881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:01:56.161126 8864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 01:01:56.161260 8864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "9e2f2c35989b42cd9f890a5776564bc5"
format_stamp: "Formatted at 2026-08-06 01:01:14 on dist-test-slave-dhph"
server_key: "9b3a0bba63de6a206fb1c8e231cbdf4b"
server_key_iv: "3139f1fd11aaefb7e7ed956a0bf4327b"
server_key_version: "encryptionkey@0"
I20260806 01:01:56.161669 8864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:01:56.179886 8864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:01:56.182777 8864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:01:56.182921 8864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:01:56.189586 8864 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:40935
I20260806 01:01:56.190048 8934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:40935 every 8 connection(s)
I20260806 01:01:56.190464 8864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:01:56.192247 8935 sys_catalog.cc:263] Verifying existing consensus state
I20260806 01:01:56.192970 8935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5: Bootstrap starting.
I20260806 01:01:56.195577 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 8864
I20260806 01:01:56.195978 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:43497
--local_ip_for_outbound_sockets=127.23.242.1
--tserver_master_addrs=127.23.242.62:40935
--webserver_port=43115
--webserver_interface=127.23.242.1
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 01:01:56.205878 8935 log.cc:824] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5: Log is configured to *not* fsync() on all Append() calls
I20260806 01:01:56.210320 8935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 01:01:56.210834 8935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5: Bootstrap complete.
I20260806 01:01:56.214699 8935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2f2c35989b42cd9f890a5776564bc5" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 40935 } }
I20260806 01:01:56.215081 8935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e2f2c35989b42cd9f890a5776564bc5, State: Initialized, Role: FOLLOWER
I20260806 01:01:56.215314 8935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2f2c35989b42cd9f890a5776564bc5" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 40935 } }
I20260806 01:01:56.215396 8935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:01:56.215443 8935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:01:56.215487 8935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [term 2 FOLLOWER]: Advancing to term 3
I20260806 01:01:56.217063 8935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2f2c35989b42cd9f890a5776564bc5" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 40935 } }
I20260806 01:01:56.217190 8935 leader_election.cc:302] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e2f2c35989b42cd9f890a5776564bc5; no voters:
I20260806 01:01:56.217396 8935 leader_election.cc:288] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 01:01:56.217507 8939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 01:01:56.217685 8935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:01:56.217705 8939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [term 3 LEADER]: Becoming Leader. State: Replica: 9e2f2c35989b42cd9f890a5776564bc5, State: Running, Role: LEADER
I20260806 01:01:56.217796 8939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2f2c35989b42cd9f890a5776564bc5" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 40935 } }
I20260806 01:01:56.218161 8940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9e2f2c35989b42cd9f890a5776564bc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2f2c35989b42cd9f890a5776564bc5" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 40935 } } }
I20260806 01:01:56.218246 8940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [sys.catalog]: This master's current role is: LEADER
I20260806 01:01:56.218475 8941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e2f2c35989b42cd9f890a5776564bc5. Latest consensus state: current_term: 3 leader_uuid: "9e2f2c35989b42cd9f890a5776564bc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e2f2c35989b42cd9f890a5776564bc5" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 40935 } } }
I20260806 01:01:56.218639 8941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5 [sys.catalog]: This master's current role is: LEADER
I20260806 01:01:56.218686 8947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:01:56.219568 8947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:01:56.219681 8947 catalog_manager.cc:1259] Loaded cluster ID: 3128c701d3a842a3b698fe3eccd0f8db
I20260806 01:01:56.219712 8947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:01:56.220942 8947 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:01:56.221143 8947 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5: Loaded TSK: 0
I20260806 01:01:56.221207 8947 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9e2f2c35989b42cd9f890a5776564bc5: Loaded TSK: 1
F20260806 01:01:56.221448 8947 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4f887ea9ad google::LogMessage::Fail() at ??:0
@ 0x7f4f887ed88a google::LogMessage::SendToLog() at ??:0
@ 0x7f4f887ea4de google::LogMessage::Flush() at ??:0
@ 0x7f4f887eba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4f8af57104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f4f8af54d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f4f907441cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f4f9074448e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f4f90744594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f4f9070ac48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f4f90714ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4f8a6b80d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4f8a6ad954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4f8b5116db start_thread at ??:0
@ 0x7f4f872e861f clone at ??:0
W20260806 01:01:56.348968 8937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:01:56.349268 8937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:01:56.349318 8937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:01:56.350769 8937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:01:56.350841 8937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:01:56.350896 8937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 01:01:56.354167 8937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:43497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=43115
--tserver_master_addrs=127.23.242.62:40935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:01:56.354952 8937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:01:56.355346 8937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:01:56.361750 8977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:01:56.361732 8978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:01:56.361732 8980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:01:56.362224 8937 server_base.cc:1034] running on GCE node
I20260806 01:01:56.362452 8937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:01:56.362839 8937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:01:56.364068 8937 hybrid_clock.cc:648] HybridClock initialized: now 1785978116364048 us; error 29 us; skew 500 ppm
Aug 06 01:01:56 dist-test-slave-dhph krb5kdc[7772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978116, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:01:56.367556 8937 init.cc:377] Logged in from keytab as kudu/127.23.242.1@KRBTEST.COM (short username kudu)
I20260806 01:01:56.368163 8937 webserver.cc:460] Webserver started at http://127.23.242.1:43115/ using document root <none> and password file <none>
I20260806 01:01:56.368468 8937 fs_manager.cc:359] Metadata directory not provided
I20260806 01:01:56.368553 8937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:01:56.370507 8937 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 01:01:56.371826 8988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:01:56.372156 8937 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 01:01:56.372303 8937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "da6c6da4a5434ab3b834d8d0eee9f57c"
format_stamp: "Formatted at 2026-08-06 01:01:14 on dist-test-slave-dhph"
server_key: "1fd76abebc85df107bb8d0b8ed92a9d2"
server_key_iv: "56379cf7796a0b50a345427f1b510d64"
server_key_version: "encryptionkey@0"
I20260806 01:01:56.372697 8937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:01:56.389957 8937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:01:56.392791 8937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:01:56.393021 8937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:01:56.393313 8937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:01:56.393880 8995 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 01:01:56.396173 8937 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 01:01:56.396281 8937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260806 01:01:56.396334 8937 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 01:01:56.397152 8937 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 01:01:56.397327 8995 tablet_bootstrap.cc:492] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c: Bootstrap starting.
I20260806 01:01:56.397370 8937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260806 01:01:56.406672 8937 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:43497
I20260806 01:01:56.406718 9106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:43497 every 8 connection(s)
I20260806 01:01:56.407157 8937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 01:01:56.413055 8995 log.cc:824] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c: Log is configured to *not* fsync() on all Append() calls
I20260806 01:01:56.415680 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 8937
I20260806 01:01:56.416074 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:41497
--local_ip_for_outbound_sockets=127.23.242.2
--tserver_master_addrs=127.23.242.62:40935
--webserver_port=38511
--webserver_interface=127.23.242.2
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 01:01:56.416577 8995 tablet_bootstrap.cc:492] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 01:01:56.416988 8995 tablet_bootstrap.cc:492] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c: Bootstrap complete.
I20260806 01:01:56.418485 8995 ts_tablet_manager.cc:1397] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.012s
I20260806 01:01:56.420192 8995 raft_consensus.cc:348] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } }
I20260806 01:01:56.420568 8995 raft_consensus.cc:729] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: da6c6da4a5434ab3b834d8d0eee9f57c, State: Initialized, Role: FOLLOWER
I20260806 01:01:56.420769 8995 consensus_queue.cc:260] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } }
I20260806 01:01:56.421164 8995 ts_tablet_manager.cc:1428] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c: Time spent starting tablet: real 0.003s user 0.000s sys 0.006s
W20260806 01:01:56.507505 9109 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:01:56.408023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.1:47991)
0806 01:01:56.408366 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:56.408411 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 01:01:56.408918 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:01:56.507287 (+ 98369us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":123,"threads_started":1}
W20260806 01:01:56.507879 9107 heartbeater.cc:644] Failed to heartbeat to 127.23.242.62:40935 (0 consecutive failures): Network error: Failed to ping master at 127.23.242.62:40935: Client connection negotiation failed: client connection to 127.23.242.62:40935: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 01:01:56.508924 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:56.508777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.1:34997)
0806 01:01:56.508834 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:56.508892 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260806 01:01:56.509382 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:56.509275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.1:57307)
0806 01:01:56.509326 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:56.509359 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 01:01:56.509522 9107 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 01:01:56.518517 9111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:01:56.518810 9111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:01:56.518837 9111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:01:56.520156 9111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:01:56.520234 9111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:01:56.520272 9111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 01:01:56.523202 9111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:41497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=38511
--tserver_master_addrs=127.23.242.62:40935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:01:56.523903 9111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:01:56.524281 9111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:01:56.531165 9120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:01:56.531165 9118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:01:56.531316 9117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:01:56.531666 9111 server_base.cc:1034] running on GCE node
I20260806 01:01:56.531991 9111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:01:56.532948 9111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:01:56.534220 9111 hybrid_clock.cc:648] HybridClock initialized: now 1785978116534186 us; error 56 us; skew 500 ppm
Aug 06 01:01:56 dist-test-slave-dhph krb5kdc[7772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978116, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:01:56.538291 9111 init.cc:377] Logged in from keytab as kudu/127.23.242.2@KRBTEST.COM (short username kudu)
I20260806 01:01:56.538894 9111 webserver.cc:460] Webserver started at http://127.23.242.2:38511/ using document root <none> and password file <none>
I20260806 01:01:56.539184 9111 fs_manager.cc:359] Metadata directory not provided
I20260806 01:01:56.539242 9111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:01:56.541174 9111 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 01:01:56.542001 9128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:01:56.542362 9111 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 01:01:56.542450 9111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "dfbb4658535f4208a4d9bdb1f31c0103"
format_stamp: "Formatted at 2026-08-06 01:01:14 on dist-test-slave-dhph"
server_key: "67f389e1ed8bdbf335c95c1873eecbb2"
server_key_iv: "1e9f00d66187292452ce4eec974ab02a"
server_key_version: "encryptionkey@0"
I20260806 01:01:56.542873 9111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:01:56.562093 9111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:01:56.565750 9111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:01:56.565910 9111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:01:56.566210 9111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:01:56.566568 9111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:01:56.566843 9111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:01:56.566872 9111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:01:56.566886 9111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:01:56.573879 9111 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:41497
I20260806 01:01:56.574079 9243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:41497 every 8 connection(s)
I20260806 01:01:56.574499 9111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 01:01:56.574818 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 9111
I20260806 01:01:56.575145 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:36385
--local_ip_for_outbound_sockets=127.23.242.3
--tserver_master_addrs=127.23.242.62:40935
--webserver_port=43959
--webserver_interface=127.23.242.3
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 01:01:56.576064 9246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:56.575479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.2:51167)
0806 01:01:56.575748 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:56.575850 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":94,"threads_started":1}
W20260806 01:01:56.576643 9244 heartbeater.cc:644] Failed to heartbeat to 127.23.242.62:40935 (0 consecutive failures): Network error: Failed to ping master at 127.23.242.62:40935: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
I20260806 01:01:56.577021 9246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:56.576898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.2:60871)
0806 01:01:56.576955 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:56.576988 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260806 01:01:56.577440 9246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:56.577351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.2:53845)
0806 01:01:56.577400 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:56.577423 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 01:01:56.577553 9244 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 01:01:56.685861 9247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:01:56.686151 9247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:01:56.686201 9247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:01:56.687701 9247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:01:56.687793 9247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:01:56.687849 9247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 01:01:56.689546 9247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:36385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=43959
--tserver_master_addrs=127.23.242.62:40935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:01:56.690052 9247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:01:56.690351 9247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:01:56.696267 9255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:01:56.696324 9247 server_base.cc:1034] running on GCE node
W20260806 01:01:56.696287 9253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:01:56.696246 9252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:01:56.696689 9247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:01:56.697077 9247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:01:56.698302 9247 hybrid_clock.cc:648] HybridClock initialized: now 1785978116698240 us; error 76 us; skew 500 ppm
Aug 06 01:01:56 dist-test-slave-dhph krb5kdc[7772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978116, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:01:56.701350 9247 init.cc:377] Logged in from keytab as kudu/127.23.242.3@KRBTEST.COM (short username kudu)
I20260806 01:01:56.701860 9247 webserver.cc:460] Webserver started at http://127.23.242.3:43959/ using document root <none> and password file <none>
I20260806 01:01:56.702116 9247 fs_manager.cc:359] Metadata directory not provided
I20260806 01:01:56.702169 9247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:01:56.703893 9247 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:01:56.704711 9262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:01:56.704988 9247 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:01:56.705133 9247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "16bd3ad6217d40e19f7a6624ad9c650b"
format_stamp: "Formatted at 2026-08-06 01:01:14 on dist-test-slave-dhph"
server_key: "cc6ba1f343637f722f44cccdb65dcc64"
server_key_iv: "3f5803827a1f305886458c8f4faaf384"
server_key_version: "encryptionkey@0"
I20260806 01:01:56.705528 9247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:01:56.714573 9247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:01:56.717115 9247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:01:56.717321 9247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:01:56.717603 9247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:01:56.717957 9247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:01:56.718017 9247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:01:56.718056 9247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:01:56.718091 9247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:01:56.725746 9247 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:36385
I20260806 01:01:56.725857 9377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:36385 every 8 connection(s)
I20260806 01:01:56.726347 9247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
I20260806 01:01:56.728854 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:56.728078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.3:60241)
0806 01:01:56.728383 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:56.728640 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":104,"threads_started":1}
W20260806 01:01:56.729470 9378 heartbeater.cc:644] Failed to heartbeat to 127.23.242.62:40935 (0 consecutive failures): Network error: Failed to ping master at 127.23.242.62:40935: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
I20260806 01:01:56.729940 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:56.729774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.3:38565)
0806 01:01:56.729875 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:56.729913 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 01:01:56.730803 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:56.730462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.3:47573)
0806 01:01:56.730648 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:56.730749 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260806 01:01:56.731056 9378 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 01:01:56.733952 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 9247
I20260806 01:01:57.510286 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:57.509974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.1:41377)
0806 01:01:57.510167 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:57.510214 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":79,"threads_started":1}
I20260806 01:01:57.578126 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:57.577906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.2:40611)
0806 01:01:57.578031 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:57.578067 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260806 01:01:57.731953 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:57.731684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.3:45757)
0806 01:01:57.731849 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:57.731891 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
I20260806 01:01:58.276517 9385 raft_consensus.cc:482] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 01:01:58.276649 9385 raft_consensus.cc:504] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } }
I20260806 01:01:58.277091 9385 leader_election.cc:288] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfbb4658535f4208a4d9bdb1f31c0103 (127.23.242.2:41497), 16bd3ad6217d40e19f7a6624ad9c650b (127.23.242.3:36385)
Aug 06 01:01:58 dist-test-slave-dhph krb5kdc[7772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978116, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for kudu/127.23.242.2@KRBTEST.COM
Aug 06 01:01:58 dist-test-slave-dhph krb5kdc[7772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978116, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for kudu/127.23.242.3@KRBTEST.COM
I20260806 01:01:58.289115 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:58.277288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:36385 (local address 127.23.242.1:44845)
0806 01:01:58.277450 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:58.277463 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 01:01:58.277534 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:01:58.278413 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:01:58.278416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:01:58.278901 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:01:58.279094 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:58.279101 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:58.280352 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:58.280353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:58.281211 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:58.281214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:58.281863 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:58.281864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:58.282005 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:58.282789 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:01:58.282794 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:01:58.284690 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:01:58.287405 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:01:58.287413 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:01:58.287415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:01:58.287634 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:01:58.287884 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:01:58.287885 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:01:58.287886 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:01:58.288021 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:01:58.288597 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:01:58.288604 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:01:58.288713 (+ 109us) client_negotiation.cc:769] Sending connection context
0806 01:01:58.288812 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 01:01:58.288995 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":61,"thread_start_us":113,"threads_started":1}
I20260806 01:01:58.289227 9386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:58.277316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:41497 (local address 127.23.242.1:46219)
0806 01:01:58.277464 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:58.277510 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 01:01:58.277573 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:01:58.278414 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:01:58.278417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:01:58.278903 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:01:58.279095 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:58.279101 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:58.280322 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:58.280323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:58.281236 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:01:58.281240 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:01:58.281898 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:01:58.281899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:01:58.281998 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:58.282760 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:01:58.282770 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:01:58.284294 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:01:58.287705 (+ 3411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:01:58.287705 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:01:58.287705 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:01:58.287931 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:01:58.288296 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:01:58.288297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:01:58.288297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:01:58.288355 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:01:58.288931 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:01:58.288932 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:01:58.288972 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 01:01:58.289044 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 01:01:58.289135 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":4,"thread_start_us":84,"threads_started":1}
I20260806 01:01:58.289286 9389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:58.277493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:44845 (local address 127.23.242.3:36385)
0806 01:01:58.277663 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 01:01:58.277678 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 01:01:58.277695 (+ 17us) server_negotiation.cc:407] Connection header received
0806 01:01:58.278172 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:01:58.278174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:01:58.278221 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:01:58.278376 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:01:58.279291 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:58.280230 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:58.281321 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:58.281762 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:58.281892 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:58.284808 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:01:58.284812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:01:58.284812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:01:58.284824 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:01:58.287303 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:01:58.287709 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:01:58.287712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:01:58.287712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:01:58.287802 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:01:58.288115 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:01:58.288117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:01:58.288117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:01:58.288510 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:01:58.288618 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 01:01:58.288911 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 01:01:58.289137 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":65,"threads_started":1}
W20260806 01:01:58.290225 8991 leader_election.cc:341] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 16bd3ad6217d40e19f7a6624ad9c650b (127.23.242.3:36385): Not found: Tablet not found: af421777b0be4879a470c36d8f3d4e0f
I20260806 01:01:58.290819 9388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:58.277440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:46219 (local address 127.23.242.2:41497)
0806 01:01:58.277628 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 01:01:58.277640 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 01:01:58.277664 (+ 24us) server_negotiation.cc:407] Connection header received
0806 01:01:58.278164 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:01:58.278167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:01:58.278211 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:01:58.278377 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:01:58.279226 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:58.280206 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:58.281362 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:01:58.281810 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:01:58.281931 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:01:58.284430 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:01:58.284468 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:01:58.284468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:01:58.284482 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:01:58.287299 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:01:58.288028 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:01:58.288031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:01:58.288031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:01:58.288195 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:01:58.288432 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:01:58.288433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:01:58.288433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:01:58.288820 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:01:58.288931 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 01:01:58.289133 (+ 202us) server_negotiation.cc:299] Negotiation successful
0806 01:01:58.290741 (+ 1608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":58,"threads_started":1}
W20260806 01:01:58.291453 8990 leader_election.cc:341] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfbb4658535f4208a4d9bdb1f31c0103 (127.23.242.2:41497): Not found: Tablet not found: af421777b0be4879a470c36d8f3d4e0f
I20260806 01:01:58.291524 8990 leader_election.cc:302] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6c6da4a5434ab3b834d8d0eee9f57c; no voters: 16bd3ad6217d40e19f7a6624ad9c650b, dfbb4658535f4208a4d9bdb1f31c0103
I20260806 01:01:58.291632 9385 raft_consensus.cc:2726] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 01:01:58.511099 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:58.510885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.1:43709)
0806 01:01:58.510986 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:58.511036 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 01:01:58.578981 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:58.578724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.2:33233)
0806 01:01:58.578897 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:58.578935 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":109,"threads_started":1}
I20260806 01:01:58.732882 9391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:58.732564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.3:39621)
0806 01:01:58.732743 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:58.732813 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":84,"threads_started":1}
I20260806 01:01:59.512238 9392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:59.511959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.1:35977)
0806 01:01:59.512142 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:59.512189 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":87,"threads_started":1}
I20260806 01:01:59.579828 9393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:59.579558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.2:52239)
0806 01:01:59.579723 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:59.579778 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":73,"threads_started":1}
I20260806 01:01:59.733834 9394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:01:59.733538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.3:40147)
0806 01:01:59.733728 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0806 01:01:59.733778 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":88,"threads_started":1}
I20260806 01:01:59.825296 9395 raft_consensus.cc:482] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 01:01:59.825431 9395 raft_consensus.cc:504] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } }
I20260806 01:01:59.825750 9395 leader_election.cc:288] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfbb4658535f4208a4d9bdb1f31c0103 (127.23.242.2:41497), 16bd3ad6217d40e19f7a6624ad9c650b (127.23.242.3:36385)
W20260806 01:01:59.826375 8990 leader_election.cc:341] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfbb4658535f4208a4d9bdb1f31c0103 (127.23.242.2:41497): Not found: Tablet not found: af421777b0be4879a470c36d8f3d4e0f
W20260806 01:01:59.826525 8991 leader_election.cc:341] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 16bd3ad6217d40e19f7a6624ad9c650b (127.23.242.3:36385): Not found: Tablet not found: af421777b0be4879a470c36d8f3d4e0f
I20260806 01:01:59.826653 8991 leader_election.cc:302] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6c6da4a5434ab3b834d8d0eee9f57c; no voters: 16bd3ad6217d40e19f7a6624ad9c650b, dfbb4658535f4208a4d9bdb1f31c0103
I20260806 01:01:59.826802 9395 raft_consensus.cc:2726] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 01:02:00.513223 9396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:00.512937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.1:47797)
0806 01:02:00.513127 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:00.513176 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":93,"threads_started":1}
I20260806 01:02:00.580909 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:00.580611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.2:48119)
0806 01:02:00.580812 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:00.580859 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":93,"threads_started":1}
I20260806 01:02:00.734881 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:00.734525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.3:44377)
0806 01:02:00.734790 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:00.734831 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":175,"threads_started":1}
I20260806 01:02:01.345628 9399 raft_consensus.cc:482] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 01:02:01.345803 9399 raft_consensus.cc:504] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } }
I20260806 01:02:01.346014 9399 leader_election.cc:288] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfbb4658535f4208a4d9bdb1f31c0103 (127.23.242.2:41497), 16bd3ad6217d40e19f7a6624ad9c650b (127.23.242.3:36385)
W20260806 01:02:01.346711 8991 leader_election.cc:341] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 16bd3ad6217d40e19f7a6624ad9c650b (127.23.242.3:36385): Not found: Tablet not found: af421777b0be4879a470c36d8f3d4e0f
W20260806 01:02:01.346863 8990 leader_election.cc:341] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfbb4658535f4208a4d9bdb1f31c0103 (127.23.242.2:41497): Not found: Tablet not found: af421777b0be4879a470c36d8f3d4e0f
I20260806 01:02:01.346935 8990 leader_election.cc:302] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6c6da4a5434ab3b834d8d0eee9f57c; no voters: 16bd3ad6217d40e19f7a6624ad9c650b, dfbb4658535f4208a4d9bdb1f31c0103
I20260806 01:02:01.347054 9399 raft_consensus.cc:2726] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 01:02:01.514436 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:01.514084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.1:41445)
0806 01:02:01.514333 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:01.514386 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":75,"threads_started":1}
I20260806 01:02:01.582129 9401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:01.581805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.2:54835)
0806 01:02:01.581991 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:01.582054 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":77,"threads_started":1}
I20260806 01:02:01.735714 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:01.735427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.3:36829)
0806 01:02:01.735615 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:01.735656 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
I20260806 01:02:02.515463 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:02.515167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.1:58717)
0806 01:02:02.515362 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:02.515412 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":102,"threads_started":1}
I20260806 01:02:02.583220 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:02.582938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.2:39515)
0806 01:02:02.583100 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:02.583154 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":70,"threads_started":1}
I20260806 01:02:02.736656 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:02.736393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.3:58193)
0806 01:02:02.736559 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:02.736606 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":73,"threads_started":1}
I20260806 01:02:03.516431 9406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:03.516120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.1:56803)
0806 01:02:03.516329 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:03.516381 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":92,"threads_started":1}
I20260806 01:02:03.584324 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:03.584018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.2:53443)
0806 01:02:03.584232 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:03.584275 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":104,"threads_started":1}
I20260806 01:02:03.737721 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:03.737293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.3:34097)
0806 01:02:03.737551 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:03.737630 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":78,"threads_started":1}
I20260806 01:02:04.287734 9409 raft_consensus.cc:482] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 01:02:04.287925 9409 raft_consensus.cc:504] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfbb4658535f4208a4d9bdb1f31c0103" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41497 } } peers { permanent_uuid: "16bd3ad6217d40e19f7a6624ad9c650b" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 36385 } } peers { permanent_uuid: "da6c6da4a5434ab3b834d8d0eee9f57c" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 43497 } }
I20260806 01:02:04.288188 9409 leader_election.cc:288] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dfbb4658535f4208a4d9bdb1f31c0103 (127.23.242.2:41497), 16bd3ad6217d40e19f7a6624ad9c650b (127.23.242.3:36385)
W20260806 01:02:04.288546 8990 leader_election.cc:341] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dfbb4658535f4208a4d9bdb1f31c0103 (127.23.242.2:41497): Not found: Tablet not found: af421777b0be4879a470c36d8f3d4e0f
W20260806 01:02:04.288817 8991 leader_election.cc:341] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 16bd3ad6217d40e19f7a6624ad9c650b (127.23.242.3:36385): Not found: Tablet not found: af421777b0be4879a470c36d8f3d4e0f
I20260806 01:02:04.288914 8991 leader_election.cc:302] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: da6c6da4a5434ab3b834d8d0eee9f57c; no voters: 16bd3ad6217d40e19f7a6624ad9c650b, dfbb4658535f4208a4d9bdb1f31c0103
I20260806 01:02:04.289017 9409 raft_consensus.cc:2726] T af421777b0be4879a470c36d8f3d4e0f P da6c6da4a5434ab3b834d8d0eee9f57c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 01:02:04.517680 9410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:04.517305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.1:43073)
0806 01:02:04.517531 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:04.517601 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":84,"threads_started":1}
I20260806 01:02:04.585539 9411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:04.585219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.2:58173)
0806 01:02:04.585424 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:04.585479 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":108,"threads_started":1}
I20260806 01:02:04.739223 9412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:04.738866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.3:45637)
0806 01:02:04.739045 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:04.739091 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":65,"threads_started":1}
I20260806 01:02:05.518735 9413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:05.518371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.1:44341)
0806 01:02:05.518535 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:05.518681 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":69,"threads_started":1}
I20260806 01:02:05.586686 9414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:05.586302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.2:42917)
0806 01:02:05.586485 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:05.586532 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":91,"threads_started":1}
I20260806 01:02:05.740586 9416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:05.740050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.3:54423)
0806 01:02:05.740268 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:05.740486 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":81,"threads_started":1}
I20260806 01:02:06.519778 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:06.519467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.1:56645)
0806 01:02:06.519687 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:06.519724 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":81,"threads_started":1}
I20260806 01:02:06.587455 9418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:06.587195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.2:58481)
0806 01:02:06.587363 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:06.587405 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":71,"threads_started":1}
I20260806 01:02:06.741757 9419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:06.741309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.3:59145)
0806 01:02:06.741589 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:06.741680 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":73,"threads_started":1}
I20260806 01:02:07.520948 9420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:07.520573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.1:57837)
0806 01:02:07.520825 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:07.520893 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":111,"threads_started":1}
I20260806 01:02:07.588428 9421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:07.588147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.2:40301)
0806 01:02:07.588335 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:07.588378 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":89,"threads_started":1}
I20260806 01:02:07.743057 9422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:07.742512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.3:48527)
0806 01:02:07.742781 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:07.742828 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":129,"threads_started":1}
I20260806 01:02:08.522145 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:08.521780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.1:46611)
0806 01:02:08.522018 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:08.522080 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":80,"threads_started":1}
I20260806 01:02:08.589442 9424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:08.589067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.2:38699)
0806 01:02:08.589287 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:08.589340 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":75,"threads_started":1}
I20260806 01:02:08.743876 9425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:08.743612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.3:43587)
0806 01:02:08.743778 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:08.743820 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":70,"threads_started":1}
I20260806 01:02:09.523519 9426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:09.523070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.1:37585)
0806 01:02:09.523421 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:09.523467 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":254,"threads_started":1}
I20260806 01:02:09.590777 9427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:09.590397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.2:40411)
0806 01:02:09.590674 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:09.590724 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":98,"threads_started":1}
I20260806 01:02:09.745141 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:09.744786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.3:44419)
0806 01:02:09.744954 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:09.745087 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":73,"threads_started":1}
I20260806 01:02:10.524896 9429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:10.524590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.1:38755)
0806 01:02:10.524778 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:10.524829 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":105,"threads_started":1}
I20260806 01:02:10.591725 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:10.591466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.2:45019)
0806 01:02:10.591632 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:10.591677 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":83,"threads_started":1}
I20260806 01:02:10.746649 9431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:10.746085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.3:43285)
0806 01:02:10.746242 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:10.746532 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":104,"threads_started":1}
I20260806 01:02:11.525936 9432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:11.525661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.1:57745)
0806 01:02:11.525830 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:11.525875 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":115,"threads_started":1}
I20260806 01:02:11.592943 9433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:11.592569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.2:46697)
0806 01:02:11.592822 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:11.592877 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":109,"threads_started":1}
I20260806 01:02:11.743810 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 8937
I20260806 01:02:11.747905 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:11.747622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:40935 (local address 127.23.242.3:40293)
0806 01:02:11.747805 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:11.747852 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.62:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":125,"threads_started":1}
I20260806 01:02:11.752085 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 9111
I20260806 01:02:11.758484 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 9247
2026-08-06T01:02:11Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59044 ms)
[----------] 30 tests from SecurityITest (261779 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 01:02:11 dist-test-slave-dhph krb5kdc[9437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 01:02:11 dist-test-slave-dhph krb5kdc[9437](info): set up 2 sockets
Aug 06 01:02:11 dist-test-slave-dhph krb5kdc[9437](info): commencing operation
krb5kdc: starting...
W20260806 01:02:12.949774 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.006s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 01:02:13 dist-test-slave-dhph krb5kdc[9437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T01:02:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:02:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 01:02:13.125101 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:43963
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:43963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:13.220885 9451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:13.221187 9451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:13.221233 9451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:13.222640 9451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:13.222720 9451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:13.222754 9451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:13.222790 9451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:13.222819 9451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:13.224454 9451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:43963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:43963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:13.224776 9451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:13.225028 9451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:13.232115 9456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:13.232095 9457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:13.232124 9459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:13.232594 9451 server_base.cc:1034] running on GCE node
I20260806 01:02:13.233214 9451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:13.233590 9451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:13.235152 9451 hybrid_clock.cc:648] HybridClock initialized: now 1785978133235014 us; error 149 us; skew 500 ppm
Aug 06 01:02:13 dist-test-slave-dhph krb5kdc[9437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978133, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:13.239646 9451 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 01:02:13.240267 9451 webserver.cc:460] Webserver started at http://127.23.242.62:32817/ using document root <none> and password file <none>
I20260806 01:02:13.240556 9451 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:13.240630 9451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:13.240795 9451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:13.241993 9451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "cbf1dfa4359a4824890fb5a99850d2c7"
format_stamp: "Formatted at 2026-08-06 01:02:13 on dist-test-slave-dhph"
server_key: "f1714ed8cd994d45fd441e684d93fcb7"
server_key_iv: "b64945767065e4be7a9e2ebff4c77a75"
server_key_version: "encryptionkey@0"
I20260806 01:02:13.242396 9451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "cbf1dfa4359a4824890fb5a99850d2c7"
format_stamp: "Formatted at 2026-08-06 01:02:13 on dist-test-slave-dhph"
server_key: "f1714ed8cd994d45fd441e684d93fcb7"
server_key_iv: "b64945767065e4be7a9e2ebff4c77a75"
server_key_version: "encryptionkey@0"
I20260806 01:02:13.244339 9451 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260806 01:02:13.245803 9466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:13.246161 9451 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 01:02:13.246308 9451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "cbf1dfa4359a4824890fb5a99850d2c7"
format_stamp: "Formatted at 2026-08-06 01:02:13 on dist-test-slave-dhph"
server_key: "f1714ed8cd994d45fd441e684d93fcb7"
server_key_iv: "b64945767065e4be7a9e2ebff4c77a75"
server_key_version: "encryptionkey@0"
I20260806 01:02:13.246423 9451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:13.266933 9451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:13.269836 9451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:13.270057 9451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:13.275682 9451 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:43963
I20260806 01:02:13.275719 9518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:43963 every 8 connection(s)
I20260806 01:02:13.276139 9451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:02:13.277529 9519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:13.280665 9519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf1dfa4359a4824890fb5a99850d2c7: Bootstrap starting.
I20260806 01:02:13.281395 9519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbf1dfa4359a4824890fb5a99850d2c7: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:13.281770 9519 log.cc:824] T 00000000000000000000000000000000 P cbf1dfa4359a4824890fb5a99850d2c7: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:13.282892 9519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf1dfa4359a4824890fb5a99850d2c7: No bootstrap required, opened a new log
I20260806 01:02:13.283177 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 9451
I20260806 01:02:13.283350 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:02:13.283555 24520 external_mini_cluster.cc:1428] Setting key db5b64f2e7b3676fd76e344267b9d69d
I20260806 01:02:13.286049 9519 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbf1dfa4359a4824890fb5a99850d2c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf1dfa4359a4824890fb5a99850d2c7" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43963 } }
I20260806 01:02:13.286298 9519 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbf1dfa4359a4824890fb5a99850d2c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:13.286360 9519 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbf1dfa4359a4824890fb5a99850d2c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbf1dfa4359a4824890fb5a99850d2c7, State: Initialized, Role: FOLLOWER
I20260806 01:02:13.286507 9519 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbf1dfa4359a4824890fb5a99850d2c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf1dfa4359a4824890fb5a99850d2c7" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43963 } }
I20260806 01:02:13.286677 9519 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbf1dfa4359a4824890fb5a99850d2c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:02:13.286757 9519 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbf1dfa4359a4824890fb5a99850d2c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:02:13.286814 9519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbf1dfa4359a4824890fb5a99850d2c7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:13.287899 9519 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbf1dfa4359a4824890fb5a99850d2c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf1dfa4359a4824890fb5a99850d2c7" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43963 } }
I20260806 01:02:13.288044 9519 leader_election.cc:302] T 00000000000000000000000000000000 P cbf1dfa4359a4824890fb5a99850d2c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbf1dfa4359a4824890fb5a99850d2c7; no voters:
I20260806 01:02:13.288259 9519 leader_election.cc:288] T 00000000000000000000000000000000 P cbf1dfa4359a4824890fb5a99850d2c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:02:13.288344 9524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbf1dfa4359a4824890fb5a99850d2c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:13.288563 9519 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbf1dfa4359a4824890fb5a99850d2c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:13.288599 9524 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbf1dfa4359a4824890fb5a99850d2c7 [term 1 LEADER]: Becoming Leader. State: Replica: cbf1dfa4359a4824890fb5a99850d2c7, State: Running, Role: LEADER
I20260806 01:02:13.288753 9524 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbf1dfa4359a4824890fb5a99850d2c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf1dfa4359a4824890fb5a99850d2c7" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43963 } }
I20260806 01:02:13.289134 9524 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf1dfa4359a4824890fb5a99850d2c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbf1dfa4359a4824890fb5a99850d2c7. Latest consensus state: current_term: 1 leader_uuid: "cbf1dfa4359a4824890fb5a99850d2c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf1dfa4359a4824890fb5a99850d2c7" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43963 } } }
I20260806 01:02:13.289217 9524 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf1dfa4359a4824890fb5a99850d2c7 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:13.289644 9532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:02:13.290333 9532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:02:13.290706 9525 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf1dfa4359a4824890fb5a99850d2c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbf1dfa4359a4824890fb5a99850d2c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf1dfa4359a4824890fb5a99850d2c7" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43963 } } }
I20260806 01:02:13.290875 9525 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf1dfa4359a4824890fb5a99850d2c7 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:13.293252 9532 catalog_manager.cc:1347] Generated new cluster ID: 06efc078b32c4d078a419c56c6c44eab
I20260806 01:02:13.293309 9532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 01:02:13 dist-test-slave-dhph krb5kdc[9437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:02:13.299904 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:13.284359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42358 (local address 127.23.242.62:43963)
0806 01:02:13.284571 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:13.284578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:13.284610 (+ 32us) server_negotiation.cc:407] Connection header received
0806 01:02:13.285980 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:13.285989 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:13.286060 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:13.286209 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:13.287137 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:13.288890 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:13.290147 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:13.290881 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:13.291331 (+ 450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:13.294454 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:13.294459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:13.294460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:13.294474 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:13.296930 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:13.297436 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:13.297440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:13.297441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:13.298151 (+ 710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:13.298463 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:13.298464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:13.298464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:13.298857 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:13.299434 (+ 577us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:13.299516 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 01:02:13.299666 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
I20260806 01:02:13.302662 9532 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:02:13.303334 9532 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:02:13.315593 9532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbf1dfa4359a4824890fb5a99850d2c7: Generated new TSK 0
I20260806 01:02:13.316298 9532 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.242.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
I20260806 01:02:13.366506 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:0
--local_ip_for_outbound_sockets=127.23.242.1
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:43963
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:13.489163 9546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:13.489567 9546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:13.489603 9546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:13.491284 9546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:13.491350 9546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:13.491384 9546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 01:02:13.493165 9546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:43963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:13.493422 9546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:13.493656 9546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:13.500572 9552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:13.500599 9554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:13.500605 9551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:13.501170 9546 server_base.cc:1034] running on GCE node
I20260806 01:02:13.501461 9546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:13.501780 9546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:13.502978 9546 hybrid_clock.cc:648] HybridClock initialized: now 1785978133502923 us; error 70 us; skew 500 ppm
Aug 06 01:02:13 dist-test-slave-dhph krb5kdc[9437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978133, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:13.505889 9546 init.cc:377] Logged in from keytab as kudu/127.23.242.1@KRBTEST.COM (short username kudu)
I20260806 01:02:13.506421 9546 webserver.cc:460] Webserver started at http://127.23.242.1:43837/ using document root <none> and password file <none>
I20260806 01:02:13.506709 9546 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:13.506767 9546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:13.506891 9546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:13.508004 9546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-0/data/instance:
uuid: "5a1a12bcd71740f0a5785099e44f4b21"
format_stamp: "Formatted at 2026-08-06 01:02:13 on dist-test-slave-dhph"
server_key: "0a2770985164cdf9b8e2e4c3791858b3"
server_key_iv: "326ad6841096560f74c083783e9fef32"
server_key_version: "encryptionkey@0"
I20260806 01:02:13.508402 9546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance:
uuid: "5a1a12bcd71740f0a5785099e44f4b21"
format_stamp: "Formatted at 2026-08-06 01:02:13 on dist-test-slave-dhph"
server_key: "0a2770985164cdf9b8e2e4c3791858b3"
server_key_iv: "326ad6841096560f74c083783e9fef32"
server_key_version: "encryptionkey@0"
I20260806 01:02:13.510480 9546 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260806 01:02:13.511636 9562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:13.511971 9546 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 01:02:13.512102 9546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "5a1a12bcd71740f0a5785099e44f4b21"
format_stamp: "Formatted at 2026-08-06 01:02:13 on dist-test-slave-dhph"
server_key: "0a2770985164cdf9b8e2e4c3791858b3"
server_key_iv: "326ad6841096560f74c083783e9fef32"
server_key_version: "encryptionkey@0"
I20260806 01:02:13.512192 9546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:13.538506 9546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:13.541916 9546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:13.542068 9546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:13.542500 9546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:02:13.542961 9546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:02:13.543023 9546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:13.543051 9546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:02:13.543066 9546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:13.552644 9546 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:46429
I20260806 01:02:13.552788 9678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:46429 every 8 connection(s)
I20260806 01:02:13.553265 9546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 01:02:13.555730 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 9546
I20260806 01:02:13.556041 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance
I20260806 01:02:13.556295 24520 external_mini_cluster.cc:1428] Setting key 200d5ab27b4ee7d392c8cee953327299
Aug 06 01:02:13 dist-test-slave-dhph krb5kdc[9437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978133, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.242.2@KRBTEST.COM; defaulting to no policy
I20260806 01:02:13.568637 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:13.554538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:48353 (local address 127.23.242.62:43963)
0806 01:02:13.554702 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:13.554705 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:13.554908 (+ 203us) server_negotiation.cc:407] Connection header received
0806 01:02:13.555594 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:13.555595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:13.555616 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:13.555683 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:13.557053 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:13.557626 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:13.559181 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:13.559552 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:13.560011 (+ 459us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:13.564956 (+ 4945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:13.564960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:13.564961 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:13.564976 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:13.566804 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:13.567346 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:13.567348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:13.567348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:13.567415 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:13.567776 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:13.567777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:13.567777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:13.567938 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:13.568045 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:13.568421 (+ 376us) server_negotiation.cc:299] Negotiation successful
0806 01:02:13.568529 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260806 01:02:13.568862 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:13.554445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43963 (local address 127.23.242.1:48353)
0806 01:02:13.554743 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:13.554799 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:13.555409 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:13.555775 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:13.555780 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:13.556655 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:13.556893 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:13.556899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:13.557762 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:13.557763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:13.559049 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:13.559053 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:13.560037 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:13.560039 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:13.560209 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:13.560919 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:02:13.560933 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:13.564799 (+ 3866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:13.566924 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:13.566936 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:13.566939 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:13.567239 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:13.567517 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:13.567518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:13.567519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:13.567683 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:13.568067 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:13.568075 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:13.568249 (+ 174us) client_negotiation.cc:769] Sending connection context
0806 01:02:13.568379 (+ 130us) client_negotiation.cc:240] Negotiation successful
0806 01:02:13.568579 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":128,"threads_started":1}
I20260806 01:02:13.570253 9679 heartbeater.cc:347] Connected to a master server at 127.23.242.62:43963
I20260806 01:02:13.570413 9679 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.2@KRBTEST.COM" created.
I20260806 01:02:13.571070 9679 heartbeater.cc:511] Master 127.23.242.62:43963 requested a full tablet report, sending...
I20260806 01:02:13.572273 9483 ts_manager.cc:194] Registered new tserver with Master: 5a1a12bcd71740f0a5785099e44f4b21 (127.23.242.1:46429)
I20260806 01:02:13.573109 9483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.1@KRBTEST.COM'} at 127.23.242.1:48353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
I20260806 01:02:13.598877 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:0
--local_ip_for_outbound_sockets=127.23.242.2
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:43963
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:13.720402 9685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:13.720659 9685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:13.720688 9685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:13.721967 9685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:13.722011 9685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:13.722040 9685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 01:02:13.723762 9685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:43963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:13.724038 9685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:13.724255 9685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:13.730973 9691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:13.730958 9693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:13.730962 9690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:13.731446 9685 server_base.cc:1034] running on GCE node
I20260806 01:02:13.731703 9685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:13.731951 9685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:13.733126 9685 hybrid_clock.cc:648] HybridClock initialized: now 1785978133733083 us; error 59 us; skew 500 ppm
Aug 06 01:02:13 dist-test-slave-dhph krb5kdc[9437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978133, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:13.737946 9685 init.cc:377] Logged in from keytab as kudu/127.23.242.2@KRBTEST.COM (short username kudu)
I20260806 01:02:13.738552 9685 webserver.cc:460] Webserver started at http://127.23.242.2:44819/ using document root <none> and password file <none>
I20260806 01:02:13.738900 9685 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:13.738981 9685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:13.739141 9685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:13.740362 9685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-1/data/instance:
uuid: "bc60379851934a1489e331940f4c6ef7"
format_stamp: "Formatted at 2026-08-06 01:02:13 on dist-test-slave-dhph"
server_key: "b6dcd94cd8b05070e58373540517b116"
server_key_iv: "a1d258ac0bffe14eb7216541064d00f2"
server_key_version: "encryptionkey@0"
I20260806 01:02:13.740778 9685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance:
uuid: "bc60379851934a1489e331940f4c6ef7"
format_stamp: "Formatted at 2026-08-06 01:02:13 on dist-test-slave-dhph"
server_key: "b6dcd94cd8b05070e58373540517b116"
server_key_iv: "a1d258ac0bffe14eb7216541064d00f2"
server_key_version: "encryptionkey@0"
I20260806 01:02:13.742353 9685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 01:02:13.743871 9701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:13.744282 9685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 01:02:13.744421 9685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "bc60379851934a1489e331940f4c6ef7"
format_stamp: "Formatted at 2026-08-06 01:02:13 on dist-test-slave-dhph"
server_key: "b6dcd94cd8b05070e58373540517b116"
server_key_iv: "a1d258ac0bffe14eb7216541064d00f2"
server_key_version: "encryptionkey@0"
I20260806 01:02:13.744520 9685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:13.760874 9685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:13.764541 9685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:13.764689 9685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:13.765045 9685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:02:13.765442 9685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:02:13.765513 9685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:13.765565 9685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:02:13.765597 9685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:13.773267 9685 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:43377
I20260806 01:02:13.773365 9819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:43377 every 8 connection(s)
I20260806 01:02:13.773900 9685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 01:02:13.777977 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 9685
I20260806 01:02:13.778115 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance
I20260806 01:02:13.778383 24520 external_mini_cluster.cc:1428] Setting key 9cf6f366f29a7a5acfa9597e2f3d9b3c
Aug 06 01:02:13 dist-test-slave-dhph krb5kdc[9437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978133, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.242.3@KRBTEST.COM; defaulting to no policy
I20260806 01:02:13.791006 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:13.775963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:53465 (local address 127.23.242.62:43963)
0806 01:02:13.776057 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:13.776060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:13.776450 (+ 390us) server_negotiation.cc:407] Connection header received
0806 01:02:13.777054 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:13.777056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:13.777073 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:13.777170 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:13.778410 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:13.779333 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:13.781230 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:13.781503 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:13.781604 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:13.785786 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:13.785788 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:13.785789 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:13.785797 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:13.788692 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:13.789278 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:13.789279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:13.789279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:13.789347 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:13.789756 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:13.789757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:13.789757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:13.789895 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:13.790112 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:13.790725 (+ 613us) server_negotiation.cc:299] Negotiation successful
0806 01:02:13.790830 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 01:02:13.791294 9822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:13.775941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43963 (local address 127.23.242.2:53465)
0806 01:02:13.776283 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:13.776326 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:13.776914 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:13.777246 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:13.777250 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:13.777950 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:13.778302 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:13.778307 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:13.779480 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:13.779482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:13.781036 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:13.781041 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:13.781629 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:13.781630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:13.781819 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:13.782558 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:02:13.782568 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:13.785551 (+ 2983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:13.788804 (+ 3253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:13.788816 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:13.788820 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:13.789175 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:13.789423 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:13.789424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:13.789424 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:13.789583 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:13.790100 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:13.790109 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:13.790494 (+ 385us) client_negotiation.cc:769] Sending connection context
0806 01:02:13.790685 (+ 191us) client_negotiation.cc:240] Negotiation successful
0806 01:02:13.790999 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":228,"threads_started":1}
I20260806 01:02:13.792387 9820 heartbeater.cc:347] Connected to a master server at 127.23.242.62:43963
I20260806 01:02:13.792557 9820 heartbeater.cc:464] Registering TS with master...
I20260806 01:02:13.792876 9820 heartbeater.cc:511] Master 127.23.242.62:43963 requested a full tablet report, sending...
I20260806 01:02:13.793493 9483 ts_manager.cc:194] Registered new tserver with Master: bc60379851934a1489e331940f4c6ef7 (127.23.242.2:43377)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.3@KRBTEST.COM" created.
I20260806 01:02:13.794129 9483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.2@KRBTEST.COM'} at 127.23.242.2:53465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
I20260806 01:02:13.818573 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:0
--local_ip_for_outbound_sockets=127.23.242.3
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:43963
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:13.925115 9825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:13.925359 9825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:13.925385 9825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:13.926812 9825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:13.926862 9825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:13.926893 9825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 01:02:13.928479 9825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:43963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:13.928933 9825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:13.929407 9825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:13.937067 9831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:13.937101 9830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:13.937103 9825 server_base.cc:1034] running on GCE node
W20260806 01:02:13.937067 9833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:13.937996 9825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:13.938297 9825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:13.939529 9825 hybrid_clock.cc:648] HybridClock initialized: now 1785978133939488 us; error 76 us; skew 500 ppm
Aug 06 01:02:13 dist-test-slave-dhph krb5kdc[9437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978133, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:13.942986 9825 init.cc:377] Logged in from keytab as kudu/127.23.242.3@KRBTEST.COM (short username kudu)
I20260806 01:02:13.943799 9825 webserver.cc:460] Webserver started at http://127.23.242.3:35929/ using document root <none> and password file <none>
I20260806 01:02:13.944064 9825 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:13.944139 9825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:13.944303 9825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:13.945474 9825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-2/data/instance:
uuid: "685cbb92d7e544cca498fe1ef622e874"
format_stamp: "Formatted at 2026-08-06 01:02:13 on dist-test-slave-dhph"
server_key: "48cc1a52867185bcd937f210ee59f8aa"
server_key_iv: "4068388a6373a1112097047132d15167"
server_key_version: "encryptionkey@0"
I20260806 01:02:13.945887 9825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance:
uuid: "685cbb92d7e544cca498fe1ef622e874"
format_stamp: "Formatted at 2026-08-06 01:02:13 on dist-test-slave-dhph"
server_key: "48cc1a52867185bcd937f210ee59f8aa"
server_key_iv: "4068388a6373a1112097047132d15167"
server_key_version: "encryptionkey@0"
I20260806 01:02:13.947737 9825 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260806 01:02:13.949193 9840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:13.949630 9825 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:13.949805 9825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "685cbb92d7e544cca498fe1ef622e874"
format_stamp: "Formatted at 2026-08-06 01:02:13 on dist-test-slave-dhph"
server_key: "48cc1a52867185bcd937f210ee59f8aa"
server_key_iv: "4068388a6373a1112097047132d15167"
server_key_version: "encryptionkey@0"
I20260806 01:02:13.949926 9825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:13.967795 9825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:13.970441 9825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:13.970566 9825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:13.970883 9825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:02:13.971231 9825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:02:13.971267 9825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:13.971292 9825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:02:13.971310 9825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:13.978633 9825 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:40195
I20260806 01:02:13.978744 9957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:40195 every 8 connection(s)
I20260806 01:02:13.979086 9825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
Aug 06 01:02:13 dist-test-slave-dhph krb5kdc[9437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978133, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:02:13.988104 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 9825
I20260806 01:02:13.988296 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance
I20260806 01:02:13.988543 24520 external_mini_cluster.cc:1428] Setting key 62e63078ac5baf96f31dd83ac473d280
I20260806 01:02:13.990934 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:13.980310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:34905 (local address 127.23.242.62:43963)
0806 01:02:13.980394 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:13.980399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:13.980665 (+ 266us) server_negotiation.cc:407] Connection header received
0806 01:02:13.981362 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:13.981363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:13.981376 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:13.981428 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:13.982545 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:13.983092 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:13.984137 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:13.984337 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:13.984419 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:13.987083 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:13.987086 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:13.987086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:13.987094 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:13.988972 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:13.989637 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:13.989639 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:13.989639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:13.989719 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:13.990162 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:13.990164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:13.990164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:13.990317 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:13.990396 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:13.990783 (+ 387us) server_negotiation.cc:299] Negotiation successful
0806 01:02:13.990862 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 01:02:13.991207 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:13.980295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43963 (local address 127.23.242.3:34905)
0806 01:02:13.980504 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:13.980555 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:13.981190 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:13.981503 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:13.981507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:13.982210 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:13.982420 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:13.982425 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:13.983197 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:13.983198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:13.984036 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:13.984039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:13.984430 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:13.984432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:13.984568 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:13.985175 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:02:13.985184 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:13.986917 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:13.989098 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:13.989106 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:13.989108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:13.989514 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:13.989843 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:13.989844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:13.989844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:13.989989 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:13.990415 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:13.990422 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:13.990561 (+ 139us) client_negotiation.cc:769] Sending connection context
0806 01:02:13.990749 (+ 188us) client_negotiation.cc:240] Negotiation successful
0806 01:02:13.990903 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":91,"threads_started":1}
I20260806 01:02:13.992012 9958 heartbeater.cc:347] Connected to a master server at 127.23.242.62:43963
I20260806 01:02:13.992205 9958 heartbeater.cc:464] Registering TS with master...
I20260806 01:02:13.992559 9958 heartbeater.cc:511] Master 127.23.242.62:43963 requested a full tablet report, sending...
I20260806 01:02:13.993204 9483 ts_manager.cc:194] Registered new tserver with Master: 685cbb92d7e544cca498fe1ef622e874 (127.23.242.3:40195)
I20260806 01:02:13.993685 9483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.3@KRBTEST.COM'} at 127.23.242.3:34905
I20260806 01:02:14.001374 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 01:02:14.011720 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:14.003092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42368 (local address 127.23.242.62:43963)
0806 01:02:14.003240 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:14.003242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:14.003252 (+ 10us) server_negotiation.cc:407] Connection header received
0806 01:02:14.003276 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:14.003277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:14.003288 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:14.003358 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:14.004052 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.004619 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.005907 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.006123 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.006200 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:14.007998 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:14.008000 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:14.008000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:14.008008 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:14.009795 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:14.010262 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:14.010263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:14.010264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:14.010324 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:14.010574 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:14.010575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:14.010575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:14.010817 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:14.010906 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:14.011399 (+ 493us) server_negotiation.cc:299] Negotiation successful
0806 01:02:14.011613 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260806 01:02:14.014185 9483 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 01:02:14.016580 9483 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 01:02:14.025323 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:14.019457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:46429 (local address 127.0.0.1:60558)
0806 01:02:14.019969 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:14.020014 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:14.020352 (+ 338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:14.020711 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:14.020714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:14.020716 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:14.020945 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:14.020949 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.022412 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.022413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:14.023686 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:14.023690 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.024951 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.024952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:14.025044 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:14.025061 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 01:02:14.025132 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 01:02:14.025197 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":56,"thread_start_us":97,"threads_started":1}
I20260806 01:02:14.025488 9974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:14.020060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45552 (local address 127.23.242.2:43377)
0806 01:02:14.020518 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:14.020525 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:14.021351 (+ 826us) server_negotiation.cc:407] Connection header received
0806 01:02:14.021451 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:14.021453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:14.021467 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:14.021563 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:14.021806 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.022751 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.024153 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.024659 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.024753 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:14.024821 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:14.025343 (+ 522us) server_negotiation.cc:299] Negotiation successful
0806 01:02:14.025398 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":332,"threads_started":1}
I20260806 01:02:14.025401 9973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:14.019753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60958 (local address 127.23.242.3:40195)
0806 01:02:14.021014 (+ 1261us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:14.021019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:14.021030 (+ 11us) server_negotiation.cc:407] Connection header received
0806 01:02:14.021073 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:14.021075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:14.021091 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:14.021177 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:14.021408 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.022542 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.024009 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.024955 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.025067 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:14.025126 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:14.025256 (+ 130us) server_negotiation.cc:299] Negotiation successful
0806 01:02:14.025302 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1217,"thread_start_us":147,"threads_started":1}
I20260806 01:02:14.025769 9970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:14.019628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60558 (local address 127.23.242.1:46429)
0806 01:02:14.020525 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:14.020530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:14.020541 (+ 11us) server_negotiation.cc:407] Connection header received
0806 01:02:14.020586 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:14.020588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:14.020606 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:14.020705 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:14.021073 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.022294 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.023798 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.024839 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.024931 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:14.024996 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:14.025648 (+ 652us) server_negotiation.cc:299] Negotiation successful
0806 01:02:14.025704 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":858,"thread_start_us":85,"threads_started":1}
I20260806 01:02:14.025760 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:14.019565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:40195 (local address 127.0.0.1:60958)
0806 01:02:14.020075 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:14.020089 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:14.020366 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:14.021206 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:14.021207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:14.021210 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:14.021300 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:14.021303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.022686 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.022687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:14.023914 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:14.023917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.025076 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.025078 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:14.025159 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:14.025164 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 01:02:14.025236 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 01:02:14.025290 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":16,"thread_start_us":152,"threads_started":1}
I20260806 01:02:14.025802 9971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:14.019728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:43377 (local address 127.0.0.1:45552)
0806 01:02:14.021267 (+ 1539us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:14.021277 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:14.021354 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:14.021632 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:14.021633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:14.021635 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:14.021690 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:14.021692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.022872 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.022873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:14.024060 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:14.024062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.025096 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.025097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:14.025155 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:14.025158 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 01:02:14.025203 (+ 45us) client_negotiation.cc:240] Negotiation successful
0806 01:02:14.025235 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1510,"mutex_wait_us":88,"thread_start_us":170,"threads_started":1}
I20260806 01:02:14.026893 9888 tablet_service.cc:1467] Processing CreateTablet for tablet 230a33ce3b7c4e08846f6fc44b328586 (DEFAULT_TABLE table=test-table [id=76faf976c44f40ab8ce7d93a493de3fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:02:14.026893 9612 tablet_service.cc:1467] Processing CreateTablet for tablet 230a33ce3b7c4e08846f6fc44b328586 (DEFAULT_TABLE table=test-table [id=76faf976c44f40ab8ce7d93a493de3fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:02:14.027165 9751 tablet_service.cc:1467] Processing CreateTablet for tablet 230a33ce3b7c4e08846f6fc44b328586 (DEFAULT_TABLE table=test-table [id=76faf976c44f40ab8ce7d93a493de3fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:02:14.027325 9612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 230a33ce3b7c4e08846f6fc44b328586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:14.027325 9888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 230a33ce3b7c4e08846f6fc44b328586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:14.027621 9751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 230a33ce3b7c4e08846f6fc44b328586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:14.031476 9976 tablet_bootstrap.cc:492] T 230a33ce3b7c4e08846f6fc44b328586 P 685cbb92d7e544cca498fe1ef622e874: Bootstrap starting.
I20260806 01:02:14.031761 9975 tablet_bootstrap.cc:492] T 230a33ce3b7c4e08846f6fc44b328586 P 5a1a12bcd71740f0a5785099e44f4b21: Bootstrap starting.
I20260806 01:02:14.032197 9977 tablet_bootstrap.cc:492] T 230a33ce3b7c4e08846f6fc44b328586 P bc60379851934a1489e331940f4c6ef7: Bootstrap starting.
I20260806 01:02:14.032311 9976 tablet_bootstrap.cc:654] T 230a33ce3b7c4e08846f6fc44b328586 P 685cbb92d7e544cca498fe1ef622e874: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:14.032372 9975 tablet_bootstrap.cc:654] T 230a33ce3b7c4e08846f6fc44b328586 P 5a1a12bcd71740f0a5785099e44f4b21: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:14.033042 9977 tablet_bootstrap.cc:654] T 230a33ce3b7c4e08846f6fc44b328586 P bc60379851934a1489e331940f4c6ef7: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:14.033063 9976 log.cc:824] T 230a33ce3b7c4e08846f6fc44b328586 P 685cbb92d7e544cca498fe1ef622e874: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:14.033063 9975 log.cc:824] T 230a33ce3b7c4e08846f6fc44b328586 P 5a1a12bcd71740f0a5785099e44f4b21: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:14.033834 9977 log.cc:824] T 230a33ce3b7c4e08846f6fc44b328586 P bc60379851934a1489e331940f4c6ef7: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:14.034406 9976 tablet_bootstrap.cc:492] T 230a33ce3b7c4e08846f6fc44b328586 P 685cbb92d7e544cca498fe1ef622e874: No bootstrap required, opened a new log
I20260806 01:02:14.034513 9976 ts_tablet_manager.cc:1397] T 230a33ce3b7c4e08846f6fc44b328586 P 685cbb92d7e544cca498fe1ef622e874: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 01:02:14.034677 9975 tablet_bootstrap.cc:492] T 230a33ce3b7c4e08846f6fc44b328586 P 5a1a12bcd71740f0a5785099e44f4b21: No bootstrap required, opened a new log
I20260806 01:02:14.034757 9975 ts_tablet_manager.cc:1397] T 230a33ce3b7c4e08846f6fc44b328586 P 5a1a12bcd71740f0a5785099e44f4b21: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 01:02:14.036940 9975 raft_consensus.cc:348] T 230a33ce3b7c4e08846f6fc44b328586 P 5a1a12bcd71740f0a5785099e44f4b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1a12bcd71740f0a5785099e44f4b21" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46429 } } peers { permanent_uuid: "685cbb92d7e544cca498fe1ef622e874" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 40195 } } peers { permanent_uuid: "bc60379851934a1489e331940f4c6ef7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 43377 } }
I20260806 01:02:14.036919 9976 raft_consensus.cc:348] T 230a33ce3b7c4e08846f6fc44b328586 P 685cbb92d7e544cca498fe1ef622e874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1a12bcd71740f0a5785099e44f4b21" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46429 } } peers { permanent_uuid: "685cbb92d7e544cca498fe1ef622e874" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 40195 } } peers { permanent_uuid: "bc60379851934a1489e331940f4c6ef7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 43377 } }
I20260806 01:02:14.037132 9976 raft_consensus.cc:374] T 230a33ce3b7c4e08846f6fc44b328586 P 685cbb92d7e544cca498fe1ef622e874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:14.037132 9975 raft_consensus.cc:374] T 230a33ce3b7c4e08846f6fc44b328586 P 5a1a12bcd71740f0a5785099e44f4b21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:14.037180 9976 raft_consensus.cc:729] T 230a33ce3b7c4e08846f6fc44b328586 P 685cbb92d7e544cca498fe1ef622e874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 685cbb92d7e544cca498fe1ef622e874, State: Initialized, Role: FOLLOWER
I20260806 01:02:14.037180 9975 raft_consensus.cc:729] T 230a33ce3b7c4e08846f6fc44b328586 P 5a1a12bcd71740f0a5785099e44f4b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1a12bcd71740f0a5785099e44f4b21, State: Initialized, Role: FOLLOWER
I20260806 01:02:14.037307 9975 consensus_queue.cc:260] T 230a33ce3b7c4e08846f6fc44b328586 P 5a1a12bcd71740f0a5785099e44f4b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1a12bcd71740f0a5785099e44f4b21" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46429 } } peers { permanent_uuid: "685cbb92d7e544cca498fe1ef622e874" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 40195 } } peers { permanent_uuid: "bc60379851934a1489e331940f4c6ef7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 43377 } }
I20260806 01:02:14.037307 9976 consensus_queue.cc:260] T 230a33ce3b7c4e08846f6fc44b328586 P 685cbb92d7e544cca498fe1ef622e874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1a12bcd71740f0a5785099e44f4b21" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46429 } } peers { permanent_uuid: "685cbb92d7e544cca498fe1ef622e874" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 40195 } } peers { permanent_uuid: "bc60379851934a1489e331940f4c6ef7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 43377 } }
I20260806 01:02:14.037634 9975 ts_tablet_manager.cc:1428] T 230a33ce3b7c4e08846f6fc44b328586 P 5a1a12bcd71740f0a5785099e44f4b21: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 01:02:14.037634 9976 ts_tablet_manager.cc:1428] T 230a33ce3b7c4e08846f6fc44b328586 P 685cbb92d7e544cca498fe1ef622e874: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260806 01:02:14.037670 9679 heartbeater.cc:503] Master 127.23.242.62:43963 was elected leader, sending a full tablet report...
I20260806 01:02:14.037869 9958 heartbeater.cc:503] Master 127.23.242.62:43963 was elected leader, sending a full tablet report...
I20260806 01:02:14.038951 9977 tablet_bootstrap.cc:492] T 230a33ce3b7c4e08846f6fc44b328586 P bc60379851934a1489e331940f4c6ef7: No bootstrap required, opened a new log
I20260806 01:02:14.039038 9977 ts_tablet_manager.cc:1397] T 230a33ce3b7c4e08846f6fc44b328586 P bc60379851934a1489e331940f4c6ef7: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260806 01:02:14.040767 9977 raft_consensus.cc:348] T 230a33ce3b7c4e08846f6fc44b328586 P bc60379851934a1489e331940f4c6ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1a12bcd71740f0a5785099e44f4b21" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46429 } } peers { permanent_uuid: "685cbb92d7e544cca498fe1ef622e874" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 40195 } } peers { permanent_uuid: "bc60379851934a1489e331940f4c6ef7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 43377 } }
I20260806 01:02:14.040951 9977 raft_consensus.cc:374] T 230a33ce3b7c4e08846f6fc44b328586 P bc60379851934a1489e331940f4c6ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:14.041002 9977 raft_consensus.cc:729] T 230a33ce3b7c4e08846f6fc44b328586 P bc60379851934a1489e331940f4c6ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc60379851934a1489e331940f4c6ef7, State: Initialized, Role: FOLLOWER
I20260806 01:02:14.041349 9977 consensus_queue.cc:260] T 230a33ce3b7c4e08846f6fc44b328586 P bc60379851934a1489e331940f4c6ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1a12bcd71740f0a5785099e44f4b21" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46429 } } peers { permanent_uuid: "685cbb92d7e544cca498fe1ef622e874" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 40195 } } peers { permanent_uuid: "bc60379851934a1489e331940f4c6ef7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 43377 } }
I20260806 01:02:14.041749 9977 ts_tablet_manager.cc:1428] T 230a33ce3b7c4e08846f6fc44b328586 P bc60379851934a1489e331940f4c6ef7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 01:02:14.041867 9820 heartbeater.cc:503] Master 127.23.242.62:43963 was elected leader, sending a full tablet report...
W20260806 01:02:14.054286 9681 tablet.cc:2369] T 230a33ce3b7c4e08846f6fc44b328586 P 5a1a12bcd71740f0a5785099e44f4b21: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 01:02:14.229847 9959 tablet.cc:2369] T 230a33ce3b7c4e08846f6fc44b328586 P 685cbb92d7e544cca498fe1ef622e874: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 01:02:14.268288 9981 raft_consensus.cc:482] T 230a33ce3b7c4e08846f6fc44b328586 P 5a1a12bcd71740f0a5785099e44f4b21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 01:02:14.268499 9981 raft_consensus.cc:504] T 230a33ce3b7c4e08846f6fc44b328586 P 5a1a12bcd71740f0a5785099e44f4b21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1a12bcd71740f0a5785099e44f4b21" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46429 } } peers { permanent_uuid: "685cbb92d7e544cca498fe1ef622e874" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 40195 } } peers { permanent_uuid: "bc60379851934a1489e331940f4c6ef7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 43377 } }
I20260806 01:02:14.268985 9981 leader_election.cc:288] T 230a33ce3b7c4e08846f6fc44b328586 P 5a1a12bcd71740f0a5785099e44f4b21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 685cbb92d7e544cca498fe1ef622e874 (127.23.242.3:40195), bc60379851934a1489e331940f4c6ef7 (127.23.242.2:43377)
I20260806 01:02:14.273802 9973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:14.269406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:48569 (local address 127.23.242.3:40195)
0806 01:02:14.269510 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:14.269512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:14.269649 (+ 137us) server_negotiation.cc:407] Connection header received
0806 01:02:14.269683 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:14.269684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:14.269696 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:14.269769 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:14.270077 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.270965 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.272677 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.273275 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.273351 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:14.273361 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:14.273673 (+ 312us) server_negotiation.cc:299] Negotiation successful
0806 01:02:14.273726 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 01:02:14.273802 9974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:14.269398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:42447 (local address 127.23.242.2:43377)
0806 01:02:14.269470 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:14.269473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:14.269646 (+ 173us) server_negotiation.cc:407] Connection header received
0806 01:02:14.269684 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:14.269685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:14.269698 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:14.269769 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:14.270076 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.270966 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.272677 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.273263 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.273351 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:14.273361 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:14.273673 (+ 312us) server_negotiation.cc:299] Negotiation successful
0806 01:02:14.273734 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 01:02:14.273767 9984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:14.269269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:43377 (local address 127.23.242.1:42447)
0806 01:02:14.269499 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:14.269512 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:14.269597 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:14.269821 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:14.269823 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:14.269826 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:14.269938 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:14.269941 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.271102 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.271103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:14.272510 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:14.272515 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.273407 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.273409 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:14.273507 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:14.273515 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 01:02:14.273583 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 01:02:14.273648 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":116,"thread_start_us":121,"threads_started":1}
I20260806 01:02:14.273876 9985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:14.269321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:40195 (local address 127.23.242.1:48569)
0806 01:02:14.269497 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:14.269512 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:14.269597 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:14.269821 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:14.269823 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:14.269826 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:14.269939 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:14.269942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.271102 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.271103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:14.272510 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:14.272515 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.273409 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.273410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:14.273507 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:14.273516 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 01:02:14.273583 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 01:02:14.273648 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":65,"thread_start_us":133,"threads_started":1}
I20260806 01:02:14.274237 9771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "230a33ce3b7c4e08846f6fc44b328586" candidate_uuid: "5a1a12bcd71740f0a5785099e44f4b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc60379851934a1489e331940f4c6ef7" is_pre_election: true
I20260806 01:02:14.274286 9908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "230a33ce3b7c4e08846f6fc44b328586" candidate_uuid: "5a1a12bcd71740f0a5785099e44f4b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685cbb92d7e544cca498fe1ef622e874" is_pre_election: true
I20260806 01:02:14.274451 9908 raft_consensus.cc:2454] T 230a33ce3b7c4e08846f6fc44b328586 P 685cbb92d7e544cca498fe1ef622e874 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1a12bcd71740f0a5785099e44f4b21 in term 0.
I20260806 01:02:14.274456 9771 raft_consensus.cc:2454] T 230a33ce3b7c4e08846f6fc44b328586 P bc60379851934a1489e331940f4c6ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1a12bcd71740f0a5785099e44f4b21 in term 0.
I20260806 01:02:14.274744 9565 leader_election.cc:302] T 230a33ce3b7c4e08846f6fc44b328586 P 5a1a12bcd71740f0a5785099e44f4b21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1a12bcd71740f0a5785099e44f4b21, bc60379851934a1489e331940f4c6ef7; no voters:
I20260806 01:02:14.274935 9981 raft_consensus.cc:2781] T 230a33ce3b7c4e08846f6fc44b328586 P 5a1a12bcd71740f0a5785099e44f4b21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 01:02:14.275007 9981 raft_consensus.cc:482] T 230a33ce3b7c4e08846f6fc44b328586 P 5a1a12bcd71740f0a5785099e44f4b21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 01:02:14.275025 9981 raft_consensus.cc:3037] T 230a33ce3b7c4e08846f6fc44b328586 P 5a1a12bcd71740f0a5785099e44f4b21 [term 0 FOLLOWER]: Advancing to term 1
W20260806 01:02:14.275869 9821 tablet.cc:2369] T 230a33ce3b7c4e08846f6fc44b328586 P bc60379851934a1489e331940f4c6ef7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 01:02:14.275962 9981 raft_consensus.cc:504] T 230a33ce3b7c4e08846f6fc44b328586 P 5a1a12bcd71740f0a5785099e44f4b21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1a12bcd71740f0a5785099e44f4b21" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46429 } } peers { permanent_uuid: "685cbb92d7e544cca498fe1ef622e874" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 40195 } } peers { permanent_uuid: "bc60379851934a1489e331940f4c6ef7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 43377 } }
I20260806 01:02:14.276122 9981 leader_election.cc:288] T 230a33ce3b7c4e08846f6fc44b328586 P 5a1a12bcd71740f0a5785099e44f4b21 [CANDIDATE]: Term 1 election: Requested vote from peers 685cbb92d7e544cca498fe1ef622e874 (127.23.242.3:40195), bc60379851934a1489e331940f4c6ef7 (127.23.242.2:43377)
I20260806 01:02:14.276314 9908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "230a33ce3b7c4e08846f6fc44b328586" candidate_uuid: "5a1a12bcd71740f0a5785099e44f4b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "685cbb92d7e544cca498fe1ef622e874"
I20260806 01:02:14.276312 9771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "230a33ce3b7c4e08846f6fc44b328586" candidate_uuid: "5a1a12bcd71740f0a5785099e44f4b21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc60379851934a1489e331940f4c6ef7"
I20260806 01:02:14.276400 9771 raft_consensus.cc:3037] T 230a33ce3b7c4e08846f6fc44b328586 P bc60379851934a1489e331940f4c6ef7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:14.276401 9908 raft_consensus.cc:3037] T 230a33ce3b7c4e08846f6fc44b328586 P 685cbb92d7e544cca498fe1ef622e874 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:14.277318 9908 raft_consensus.cc:2454] T 230a33ce3b7c4e08846f6fc44b328586 P 685cbb92d7e544cca498fe1ef622e874 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1a12bcd71740f0a5785099e44f4b21 in term 1.
I20260806 01:02:14.277318 9771 raft_consensus.cc:2454] T 230a33ce3b7c4e08846f6fc44b328586 P bc60379851934a1489e331940f4c6ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1a12bcd71740f0a5785099e44f4b21 in term 1.
I20260806 01:02:14.277524 9563 leader_election.cc:302] T 230a33ce3b7c4e08846f6fc44b328586 P 5a1a12bcd71740f0a5785099e44f4b21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1a12bcd71740f0a5785099e44f4b21, 685cbb92d7e544cca498fe1ef622e874; no voters:
I20260806 01:02:14.277621 9981 raft_consensus.cc:2781] T 230a33ce3b7c4e08846f6fc44b328586 P 5a1a12bcd71740f0a5785099e44f4b21 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:14.277813 9981 raft_consensus.cc:686] T 230a33ce3b7c4e08846f6fc44b328586 P 5a1a12bcd71740f0a5785099e44f4b21 [term 1 LEADER]: Becoming Leader. State: Replica: 5a1a12bcd71740f0a5785099e44f4b21, State: Running, Role: LEADER
I20260806 01:02:14.277912 9981 consensus_queue.cc:237] T 230a33ce3b7c4e08846f6fc44b328586 P 5a1a12bcd71740f0a5785099e44f4b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1a12bcd71740f0a5785099e44f4b21" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46429 } } peers { permanent_uuid: "685cbb92d7e544cca498fe1ef622e874" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 40195 } } peers { permanent_uuid: "bc60379851934a1489e331940f4c6ef7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 43377 } }
I20260806 01:02:14.278839 9483 catalog_manager.cc:5495] T 230a33ce3b7c4e08846f6fc44b328586 P 5a1a12bcd71740f0a5785099e44f4b21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a1a12bcd71740f0a5785099e44f4b21 (127.23.242.1). New cstate: current_term: 1 leader_uuid: "5a1a12bcd71740f0a5785099e44f4b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1a12bcd71740f0a5785099e44f4b21" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "685cbb92d7e544cca498fe1ef622e874" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc60379851934a1489e331940f4c6ef7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 43377 } health_report { overall_health: UNKNOWN } } }
I20260806 01:02:14.313333 9970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:14.309527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.23.242.1:46429)
0806 01:02:14.309695 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:14.309698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:14.309710 (+ 12us) server_negotiation.cc:407] Connection header received
0806 01:02:14.309757 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:14.309758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:14.309784 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:14.309875 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 01:02:14.310166 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.310717 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.311615 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.311922 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.311995 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:14.312705 (+ 710us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 01:02:14.312872 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 01:02:14.313011 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:14.313106 (+ 95us) server_negotiation.cc:299] Negotiation successful
0806 01:02:14.313149 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260806 01:02:14.316134 9771 raft_consensus.cc:1261] T 230a33ce3b7c4e08846f6fc44b328586 P bc60379851934a1489e331940f4c6ef7 [term 1 FOLLOWER]: Refusing update from remote peer 5a1a12bcd71740f0a5785099e44f4b21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:02:14.316218 9908 raft_consensus.cc:1261] T 230a33ce3b7c4e08846f6fc44b328586 P 685cbb92d7e544cca498fe1ef622e874 [term 1 FOLLOWER]: Refusing update from remote peer 5a1a12bcd71740f0a5785099e44f4b21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:02:14.316645 9981 consensus_queue.cc:1035] T 230a33ce3b7c4e08846f6fc44b328586 P 5a1a12bcd71740f0a5785099e44f4b21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc60379851934a1489e331940f4c6ef7" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 43377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:02:14.316849 9986 consensus_queue.cc:1035] T 230a33ce3b7c4e08846f6fc44b328586 P 5a1a12bcd71740f0a5785099e44f4b21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "685cbb92d7e544cca498fe1ef622e874" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:02:14.319640 9990 mvcc.cc:205] Tried to move back new op lower bound from 7315366438155292672 to 7315366438002855936. Current Snapshot: MvccSnapshot[applied={T|T < 7315366438155292672}]
I20260806 01:02:14.322227 9483 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 01:02:14.322367 9483 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 01:02:14.323429 9483 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cbf1dfa4359a4824890fb5a99850d2c7: Sending DeleteTablet for 3 replicas of tablet 230a33ce3b7c4e08846f6fc44b328586
I20260806 01:02:14.323804 9612 tablet_service.cc:1514] Processing DeleteTablet for tablet 230a33ce3b7c4e08846f6fc44b328586 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:02:14 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:60558
I20260806 01:02:14.323879 9751 tablet_service.cc:1514] Processing DeleteTablet for tablet 230a33ce3b7c4e08846f6fc44b328586 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:02:14 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:45552
I20260806 01:02:14.324272 9999 tablet_replica.cc:330] T 230a33ce3b7c4e08846f6fc44b328586 P bc60379851934a1489e331940f4c6ef7: stopping tablet replica
I20260806 01:02:14.324283 9888 tablet_service.cc:1514] Processing DeleteTablet for tablet 230a33ce3b7c4e08846f6fc44b328586 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:02:14 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:60958
I20260806 01:02:14.324424 9999 raft_consensus.cc:2229] T 230a33ce3b7c4e08846f6fc44b328586 P bc60379851934a1489e331940f4c6ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 01:02:14.324560 9999 raft_consensus.cc:2258] T 230a33ce3b7c4e08846f6fc44b328586 P bc60379851934a1489e331940f4c6ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 01:02:14.324756 10000 tablet_replica.cc:330] T 230a33ce3b7c4e08846f6fc44b328586 P 685cbb92d7e544cca498fe1ef622e874: stopping tablet replica
I20260806 01:02:14.324867 10000 raft_consensus.cc:2229] T 230a33ce3b7c4e08846f6fc44b328586 P 685cbb92d7e544cca498fe1ef622e874 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 01:02:14.324891 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 9546
I20260806 01:02:14.324985 10000 raft_consensus.cc:2258] T 230a33ce3b7c4e08846f6fc44b328586 P 685cbb92d7e544cca498fe1ef622e874 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 01:02:14.325771 10000 ts_tablet_manager.cc:1905] T 230a33ce3b7c4e08846f6fc44b328586 P 685cbb92d7e544cca498fe1ef622e874: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 01:02:14.327636 9999 ts_tablet_manager.cc:1905] T 230a33ce3b7c4e08846f6fc44b328586 P bc60379851934a1489e331940f4c6ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 01:02:14.329255 10000 ts_tablet_manager.cc:1918] T 230a33ce3b7c4e08846f6fc44b328586 P 685cbb92d7e544cca498fe1ef622e874: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 01:02:14.329360 10000 log.cc:1196] T 230a33ce3b7c4e08846f6fc44b328586 P 685cbb92d7e544cca498fe1ef622e874: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-2/wal/wals/230a33ce3b7c4e08846f6fc44b328586
I20260806 01:02:14.329753 10000 ts_tablet_manager.cc:1939] T 230a33ce3b7c4e08846f6fc44b328586 P 685cbb92d7e544cca498fe1ef622e874: Deleting consensus metadata
I20260806 01:02:14.330500 9999 ts_tablet_manager.cc:1918] T 230a33ce3b7c4e08846f6fc44b328586 P bc60379851934a1489e331940f4c6ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 01:02:14.330624 9999 log.cc:1196] T 230a33ce3b7c4e08846f6fc44b328586 P bc60379851934a1489e331940f4c6ef7: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785977869987768-24520-0/minicluster-data/ts-1/wal/wals/230a33ce3b7c4e08846f6fc44b328586
I20260806 01:02:14.330917 9999 ts_tablet_manager.cc:1939] T 230a33ce3b7c4e08846f6fc44b328586 P bc60379851934a1489e331940f4c6ef7: Deleting consensus metadata
I20260806 01:02:14.331246 9467 catalog_manager.cc:4841] TS 685cbb92d7e544cca498fe1ef622e874 (127.23.242.3:40195): tablet 230a33ce3b7c4e08846f6fc44b328586 (table test-table [id=76faf976c44f40ab8ce7d93a493de3fd]) successfully deleted
I20260806 01:02:14.331715 9469 catalog_manager.cc:4841] TS bc60379851934a1489e331940f4c6ef7 (127.23.242.2:43377): tablet 230a33ce3b7c4e08846f6fc44b328586 (table test-table [id=76faf976c44f40ab8ce7d93a493de3fd]) successfully deleted
W20260806 01:02:14.334273 9469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.242.1:46429 (error 108)
I20260806 01:02:14.334458 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 9685
I20260806 01:02:14.335716 9971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:14.335561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:46429 (local address 127.0.0.1:60568)
0806 01:02:14.335625 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:14.335667 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.1:46429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 01:02:14.335857 9469 catalog_manager.cc:4584] TS 5a1a12bcd71740f0a5785099e44f4b21 (127.23.242.1:46429): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 230a33ce3b7c4e08846f6fc44b328586: Network error: Client connection negotiation failed: client connection to 127.23.242.1:46429: connect: Connection refused (error 111)
I20260806 01:02:14.343152 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 9825
I20260806 01:02:14.349207 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 9451
2026-08-06T01:02:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2587 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-06T01:02:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:02:14Z Disabled control of system clock
I20260806 01:02:14.382447 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:43919
--webserver_interface=127.23.242.62
--webserver_port=0
--builtin_ntp_servers=127.23.242.20:33553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:43919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 01:02:14.486166 10005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:14.486455 10005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:14.486481 10005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:14.488143 10005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:14.488212 10005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:14.488229 10005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:14.488245 10005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:14.488260 10005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:14.490056 10005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:33553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:43919
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:43919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:14.490304 10005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:14.490628 10005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:14.496065 10014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:14.496065 10011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:14.496320 10012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:14.496644 10005 server_base.cc:1034] running on GCE node
I20260806 01:02:14.497010 10005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:14.497373 10005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:14.498549 10005 hybrid_clock.cc:648] HybridClock initialized: now 1785978134498524 us; error 48 us; skew 500 ppm
I20260806 01:02:14.505299 10005 webserver.cc:460] Webserver started at http://127.23.242.62:39533/ using document root <none> and password file <none>
I20260806 01:02:14.505728 10005 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:14.505795 10005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:14.505914 10005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:14.507299 10005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "1a77d5ca37274ec0b021de23af913f28"
format_stamp: "Formatted at 2026-08-06 01:02:14 on dist-test-slave-dhph"
server_key: "bfee1ceb931e00bec9f80dbc2f2bfa80"
server_key_iv: "2c16a809339ef9f710e0970611d1b6ae"
server_key_version: "encryptionkey@0"
I20260806 01:02:14.507727 10005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "1a77d5ca37274ec0b021de23af913f28"
format_stamp: "Formatted at 2026-08-06 01:02:14 on dist-test-slave-dhph"
server_key: "bfee1ceb931e00bec9f80dbc2f2bfa80"
server_key_iv: "2c16a809339ef9f710e0970611d1b6ae"
server_key_version: "encryptionkey@0"
I20260806 01:02:14.509356 10005 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 01:02:14.510496 10020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:14.510854 10005 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:14.510938 10005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "1a77d5ca37274ec0b021de23af913f28"
format_stamp: "Formatted at 2026-08-06 01:02:14 on dist-test-slave-dhph"
server_key: "bfee1ceb931e00bec9f80dbc2f2bfa80"
server_key_iv: "2c16a809339ef9f710e0970611d1b6ae"
server_key_version: "encryptionkey@0"
I20260806 01:02:14.511051 10005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:14.527273 10005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:14.527702 10005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:14.527869 10005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:14.533258 10005 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:43919
I20260806 01:02:14.533438 10074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:43919 every 8 connection(s)
I20260806 01:02:14.533807 10005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:02:14.535036 10075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:14.538915 10075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a77d5ca37274ec0b021de23af913f28: Bootstrap starting.
I20260806 01:02:14.539860 10075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a77d5ca37274ec0b021de23af913f28: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:14.540295 10075 log.cc:824] T 00000000000000000000000000000000 P 1a77d5ca37274ec0b021de23af913f28: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:14.540740 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 10005
I20260806 01:02:14.540894 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:02:14.541388 24520 external_mini_cluster.cc:1428] Setting key 95c436c1b9342a94e3d227960501d0aa
I20260806 01:02:14.541786 10075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a77d5ca37274ec0b021de23af913f28: No bootstrap required, opened a new log
I20260806 01:02:14.544389 10075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a77d5ca37274ec0b021de23af913f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a77d5ca37274ec0b021de23af913f28" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43919 } }
I20260806 01:02:14.544587 10075 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a77d5ca37274ec0b021de23af913f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:14.544639 10075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a77d5ca37274ec0b021de23af913f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a77d5ca37274ec0b021de23af913f28, State: Initialized, Role: FOLLOWER
I20260806 01:02:14.544782 10075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a77d5ca37274ec0b021de23af913f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a77d5ca37274ec0b021de23af913f28" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43919 } }
I20260806 01:02:14.544888 10075 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a77d5ca37274ec0b021de23af913f28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:02:14.544932 10075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a77d5ca37274ec0b021de23af913f28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:02:14.544983 10075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a77d5ca37274ec0b021de23af913f28 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:14.545672 10075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a77d5ca37274ec0b021de23af913f28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a77d5ca37274ec0b021de23af913f28" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43919 } }
I20260806 01:02:14.545796 10075 leader_election.cc:302] T 00000000000000000000000000000000 P 1a77d5ca37274ec0b021de23af913f28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a77d5ca37274ec0b021de23af913f28; no voters:
I20260806 01:02:14.546048 10075 leader_election.cc:288] T 00000000000000000000000000000000 P 1a77d5ca37274ec0b021de23af913f28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:02:14.546072 10080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a77d5ca37274ec0b021de23af913f28 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:14.546190 10080 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a77d5ca37274ec0b021de23af913f28 [term 1 LEADER]: Becoming Leader. State: Replica: 1a77d5ca37274ec0b021de23af913f28, State: Running, Role: LEADER
I20260806 01:02:14.546286 10080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a77d5ca37274ec0b021de23af913f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a77d5ca37274ec0b021de23af913f28" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43919 } }
I20260806 01:02:14.546365 10075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a77d5ca37274ec0b021de23af913f28 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:14.546640 10080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a77d5ca37274ec0b021de23af913f28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a77d5ca37274ec0b021de23af913f28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a77d5ca37274ec0b021de23af913f28" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43919 } } }
I20260806 01:02:14.546715 10080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a77d5ca37274ec0b021de23af913f28 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:14.546759 10081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a77d5ca37274ec0b021de23af913f28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a77d5ca37274ec0b021de23af913f28. Latest consensus state: current_term: 1 leader_uuid: "1a77d5ca37274ec0b021de23af913f28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a77d5ca37274ec0b021de23af913f28" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43919 } } }
I20260806 01:02:14.546844 10081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a77d5ca37274ec0b021de23af913f28 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:14.547349 10084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:02:14.548467 10084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:02:14.549718 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:14.542084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42114 (local address 127.23.242.62:43919)
0806 01:02:14.542524 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:14.542533 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:14.542567 (+ 34us) server_negotiation.cc:407] Connection header received
0806 01:02:14.543583 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:14.543592 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:14.543661 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:14.543840 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:14.544070 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.545551 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.546704 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.547585 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.547731 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:14.548780 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:14.548785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:14.548785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:14.549040 (+ 255us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:14.549113 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:14.549129 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:14.549199 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:14.549441 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 01:02:14.549538 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":291,"threads_started":1}
I20260806 01:02:14.551669 10084 catalog_manager.cc:1347] Generated new cluster ID: 4f83b098849749efab1ba9b5593e322b
I20260806 01:02:14.551741 10084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:02:14.562680 10084 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:02:14.563414 10084 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:02:14.570410 10084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a77d5ca37274ec0b021de23af913f28: Generated new TSK 0
I20260806 01:02:14.570684 10084 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 01:02:14.572700 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:0
--local_ip_for_outbound_sockets=127.23.242.1
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:43919
--builtin_ntp_servers=127.23.242.20:33553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 01:02:14.688485 10100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:14.688854 10100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:14.688913 10100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:14.691088 10100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:14.691169 10100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:14.691226 10100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 01:02:14.693214 10100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:33553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:43919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:14.693542 10100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:14.693936 10100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:14.701907 10109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:14.701910 10106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:14.702175 10100 server_base.cc:1034] running on GCE node
W20260806 01:02:14.701907 10107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:14.702504 10100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:14.702826 10100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:14.704547 10100 hybrid_clock.cc:648] HybridClock initialized: now 1785978134704520 us; error 38 us; skew 500 ppm
I20260806 01:02:14.706331 10100 webserver.cc:460] Webserver started at http://127.23.242.1:35789/ using document root <none> and password file <none>
I20260806 01:02:14.706703 10100 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:14.706842 10100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:14.707073 10100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:14.708240 10100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-0/data/instance:
uuid: "30f5789e0f8c456e8540bac21bff5d70"
format_stamp: "Formatted at 2026-08-06 01:02:14 on dist-test-slave-dhph"
server_key: "949a256dbc88d4bca5bbcc177d513950"
server_key_iv: "43cb3b63210e4e8d3f7136bc7134acd3"
server_key_version: "encryptionkey@0"
I20260806 01:02:14.708802 10100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance:
uuid: "30f5789e0f8c456e8540bac21bff5d70"
format_stamp: "Formatted at 2026-08-06 01:02:14 on dist-test-slave-dhph"
server_key: "949a256dbc88d4bca5bbcc177d513950"
server_key_iv: "43cb3b63210e4e8d3f7136bc7134acd3"
server_key_version: "encryptionkey@0"
I20260806 01:02:14.711200 10100 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260806 01:02:14.712473 10116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:14.712884 10100 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:14.713001 10100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "30f5789e0f8c456e8540bac21bff5d70"
format_stamp: "Formatted at 2026-08-06 01:02:14 on dist-test-slave-dhph"
server_key: "949a256dbc88d4bca5bbcc177d513950"
server_key_iv: "43cb3b63210e4e8d3f7136bc7134acd3"
server_key_version: "encryptionkey@0"
I20260806 01:02:14.713080 10100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:14.731837 10100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:14.732245 10100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:14.732404 10100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:14.732748 10100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:02:14.733232 10100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:02:14.733284 10100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:14.733325 10100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:02:14.733345 10100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:14.743471 10100 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:40939
I20260806 01:02:14.743785 10231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:40939 every 8 connection(s)
I20260806 01:02:14.744004 10100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 01:02:14.751212 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:14.745050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:57781 (local address 127.23.242.62:43919)
0806 01:02:14.745137 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:14.745140 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:14.745812 (+ 672us) server_negotiation.cc:407] Connection header received
0806 01:02:14.746986 (+ 1174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:14.746988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:14.747008 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:14.747085 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:14.747880 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.748460 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.749584 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.749793 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.749884 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:14.750878 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:14.750879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:14.750879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:14.750892 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:14.750905 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:14.750908 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:14.750969 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:14.751085 (+ 116us) server_negotiation.cc:299] Negotiation successful
0806 01:02:14.751113 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 01:02:14.751448 10234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:14.745102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43919 (local address 127.23.242.1:57781)
0806 01:02:14.745483 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:14.745552 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:14.746777 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:14.747174 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:14.747178 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:14.747188 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:14.747749 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:14.747754 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.748594 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.748595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:14.749478 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:14.749481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.749898 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.749899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:14.750047 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:14.750700 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 01:02:14.750701 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:14.750731 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 01:02:14.750757 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:14.750971 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:14.750978 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:14.750981 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 01:02:14.751073 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 01:02:14.751187 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":110,"threads_started":1}
I20260806 01:02:14.751822 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 10100
I20260806 01:02:14.751940 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance
I20260806 01:02:14.752136 10232 heartbeater.cc:347] Connected to a master server at 127.23.242.62:43919
I20260806 01:02:14.752185 24520 external_mini_cluster.cc:1428] Setting key beb00f4796a2fe968f91e63d577b137a
I20260806 01:02:14.752345 10232 heartbeater.cc:464] Registering TS with master...
I20260806 01:02:14.752666 10232 heartbeater.cc:511] Master 127.23.242.62:43919 requested a full tablet report, sending...
I20260806 01:02:14.753326 10038 ts_manager.cc:194] Registered new tserver with Master: 30f5789e0f8c456e8540bac21bff5d70 (127.23.242.1:40939)
I20260806 01:02:14.753445 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:0
--local_ip_for_outbound_sockets=127.23.242.2
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:43919
--builtin_ntp_servers=127.23.242.20:33553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 01:02:14.754341 10038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.242.1:57781
W20260806 01:02:14.849788 10235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:14.850023 10235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:14.850046 10235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:14.851362 10235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:14.851409 10235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:14.851437 10235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 01:02:14.852854 10235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:33553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:43919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:14.853119 10235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:14.853451 10235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:14.860332 10243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:14.860342 10241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:14.860333 10240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:14.860710 10235 server_base.cc:1034] running on GCE node
I20260806 01:02:14.861042 10235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:14.861344 10235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:14.862643 10235 hybrid_clock.cc:648] HybridClock initialized: now 1785978134862506 us; error 77 us; skew 500 ppm
I20260806 01:02:14.864569 10235 webserver.cc:460] Webserver started at http://127.23.242.2:44821/ using document root <none> and password file <none>
I20260806 01:02:14.864899 10235 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:14.864982 10235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:14.865141 10235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:14.866287 10235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-1/data/instance:
uuid: "019f17402f4f4b7fa68fea4ed7ecee3c"
format_stamp: "Formatted at 2026-08-06 01:02:14 on dist-test-slave-dhph"
server_key: "dcc06f4b6e5342ecf97e98b883046794"
server_key_iv: "94e84ce65ba969b2d8982aaf1f6b2d56"
server_key_version: "encryptionkey@0"
I20260806 01:02:14.866732 10235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance:
uuid: "019f17402f4f4b7fa68fea4ed7ecee3c"
format_stamp: "Formatted at 2026-08-06 01:02:14 on dist-test-slave-dhph"
server_key: "dcc06f4b6e5342ecf97e98b883046794"
server_key_iv: "94e84ce65ba969b2d8982aaf1f6b2d56"
server_key_version: "encryptionkey@0"
I20260806 01:02:14.868221 10235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 01:02:14.869464 10250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:14.869848 10235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:14.869987 10235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "019f17402f4f4b7fa68fea4ed7ecee3c"
format_stamp: "Formatted at 2026-08-06 01:02:14 on dist-test-slave-dhph"
server_key: "dcc06f4b6e5342ecf97e98b883046794"
server_key_iv: "94e84ce65ba969b2d8982aaf1f6b2d56"
server_key_version: "encryptionkey@0"
I20260806 01:02:14.870090 10235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:14.902670 10235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:14.903054 10235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:14.903234 10235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:14.903548 10235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:02:14.903942 10235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:02:14.904007 10235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:14.904091 10235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:02:14.904145 10235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:14.913903 10235 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:36365
I20260806 01:02:14.913945 10367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:36365 every 8 connection(s)
I20260806 01:02:14.914429 10235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 01:02:14.921849 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 10235
I20260806 01:02:14.922083 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance
I20260806 01:02:14.922406 24520 external_mini_cluster.cc:1428] Setting key f6ea4561447968c6d354b292a92e4dbe
I20260806 01:02:14.924467 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:14.915508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:49129 (local address 127.23.242.62:43919)
0806 01:02:14.915573 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:14.915576 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:14.915861 (+ 285us) server_negotiation.cc:407] Connection header received
0806 01:02:14.916387 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:14.916388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:14.916399 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:14.916456 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:14.917225 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.918217 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.920474 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.920703 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.920774 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:14.923522 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:14.923524 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:14.923525 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:14.923530 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:14.923552 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:14.923557 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:14.923621 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:14.924112 (+ 491us) server_negotiation.cc:299] Negotiation successful
0806 01:02:14.924173 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 01:02:14.924825 10370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:14.915516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43919 (local address 127.23.242.2:49129)
0806 01:02:14.915704 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:14.915740 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:14.916242 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:14.916560 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:14.916564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:14.916573 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:14.917097 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:14.917101 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.919143 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.919145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:14.920105 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:14.920108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:14.921395 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:14.921396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:14.921571 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:14.922529 (+ 958us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 01:02:14.922530 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:14.923344 (+ 814us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 01:02:14.923390 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:14.923677 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:14.923684 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:14.923688 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 01:02:14.923843 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 01:02:14.924086 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260806 01:02:14.925160 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:0
--local_ip_for_outbound_sockets=127.23.242.3
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:43919
--builtin_ntp_servers=127.23.242.20:33553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 01:02:14.925909 10368 heartbeater.cc:347] Connected to a master server at 127.23.242.62:43919
I20260806 01:02:14.926085 10368 heartbeater.cc:464] Registering TS with master...
I20260806 01:02:14.926487 10368 heartbeater.cc:511] Master 127.23.242.62:43919 requested a full tablet report, sending...
I20260806 01:02:14.928525 10038 ts_manager.cc:194] Registered new tserver with Master: 019f17402f4f4b7fa68fea4ed7ecee3c (127.23.242.2:36365)
I20260806 01:02:14.928998 10038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.242.2:49129
W20260806 01:02:15.033298 10371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:15.033646 10371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:15.033759 10371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:15.035223 10371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:15.035274 10371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:15.035305 10371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 01:02:15.036847 10371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:33553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:43919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:15.037165 10371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:15.037520 10371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:15.046128 10376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:15.046281 10371 server_base.cc:1034] running on GCE node
W20260806 01:02:15.046164 10377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:15.046164 10379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:15.047465 10371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:15.047842 10371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:15.049081 10371 hybrid_clock.cc:648] HybridClock initialized: now 1785978135049027 us; error 62 us; skew 500 ppm
I20260806 01:02:15.050776 10371 webserver.cc:460] Webserver started at http://127.23.242.3:32979/ using document root <none> and password file <none>
I20260806 01:02:15.051029 10371 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:15.051077 10371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:15.051208 10371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:15.052693 10371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-2/data/instance:
uuid: "13e46e15b59f4bdab76285152d690417"
format_stamp: "Formatted at 2026-08-06 01:02:15 on dist-test-slave-dhph"
server_key: "58d56904d3b8a5a79e459c1fa7f2a267"
server_key_iv: "4c1f10de7f847ba4e00974707421ea48"
server_key_version: "encryptionkey@0"
I20260806 01:02:15.053125 10371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance:
uuid: "13e46e15b59f4bdab76285152d690417"
format_stamp: "Formatted at 2026-08-06 01:02:15 on dist-test-slave-dhph"
server_key: "58d56904d3b8a5a79e459c1fa7f2a267"
server_key_iv: "4c1f10de7f847ba4e00974707421ea48"
server_key_version: "encryptionkey@0"
I20260806 01:02:15.055023 10371 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260806 01:02:15.056857 10387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:15.057090 10371 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260806 01:02:15.057194 10371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "13e46e15b59f4bdab76285152d690417"
format_stamp: "Formatted at 2026-08-06 01:02:15 on dist-test-slave-dhph"
server_key: "58d56904d3b8a5a79e459c1fa7f2a267"
server_key_iv: "4c1f10de7f847ba4e00974707421ea48"
server_key_version: "encryptionkey@0"
I20260806 01:02:15.057267 10371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:15.078672 10371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:15.079093 10371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:15.079247 10371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:15.079495 10371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:02:15.079897 10371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:02:15.079938 10371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:15.079967 10371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:02:15.079981 10371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:15.087474 10371 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:44319
I20260806 01:02:15.087595 10503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:44319 every 8 connection(s)
I20260806 01:02:15.088054 10371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
I20260806 01:02:15.093799 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:15.088987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:35499 (local address 127.23.242.62:43919)
0806 01:02:15.089127 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:15.089130 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:15.089335 (+ 205us) server_negotiation.cc:407] Connection header received
0806 01:02:15.089935 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:15.089937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:15.089950 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:15.089999 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:15.090734 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.091247 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.092246 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.092450 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.092538 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:15.093478 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:15.093480 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:15.093480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:15.093485 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:15.093500 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:15.093504 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:15.093559 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:15.093688 (+ 129us) server_negotiation.cc:299] Negotiation successful
0806 01:02:15.093715 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260806 01:02:15.094033 10506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:15.088935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43919 (local address 127.23.242.3:35499)
0806 01:02:15.089201 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:15.089237 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:15.089817 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:15.090069 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:15.090079 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:15.090088 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:15.090629 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:15.090634 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.091349 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.091350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:15.092146 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:15.092148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.092552 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.092553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:15.092694 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:15.093311 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 01:02:15.093312 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:15.093341 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 01:02:15.093364 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:15.093569 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:15.093574 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:15.093578 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 01:02:15.093673 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 01:02:15.093781 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":104,"threads_started":1}
I20260806 01:02:15.094739 10504 heartbeater.cc:347] Connected to a master server at 127.23.242.62:43919
I20260806 01:02:15.094833 10504 heartbeater.cc:464] Registering TS with master...
I20260806 01:02:15.095039 10504 heartbeater.cc:511] Master 127.23.242.62:43919 requested a full tablet report, sending...
I20260806 01:02:15.095588 10038 ts_manager.cc:194] Registered new tserver with Master: 13e46e15b59f4bdab76285152d690417 (127.23.242.3:44319)
I20260806 01:02:15.096094 10038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.242.3:35499
I20260806 01:02:15.096412 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 10371
I20260806 01:02:15.096520 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance
I20260806 01:02:15.096728 24520 external_mini_cluster.cc:1428] Setting key 72ff432ef9928f8db46fb6358dd8884d
I20260806 01:02:15.097926 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 01:02:15.105495 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:15.100269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42130 (local address 127.23.242.62:43919)
0806 01:02:15.100420 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:15.100422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:15.100821 (+ 399us) server_negotiation.cc:407] Connection header received
0806 01:02:15.101292 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:15.101293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:15.101308 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:15.101374 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:15.101680 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.102235 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.103381 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.103580 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.103673 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:15.104966 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:15.104967 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:15.104968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:15.104973 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:15.104988 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:15.104992 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:15.105057 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:15.105395 (+ 338us) server_negotiation.cc:299] Negotiation successful
0806 01:02:15.105423 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260806 01:02:15.108534 10038 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 01:02:15.110402 10038 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 01:02:15.118327 10520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:15.114471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50560 (local address 127.23.242.1:40939)
0806 01:02:15.114699 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:15.114704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:15.114715 (+ 11us) server_negotiation.cc:407] Connection header received
0806 01:02:15.114759 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:15.114761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:15.114777 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:15.114856 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:15.115053 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.115972 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.117197 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.117901 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.117978 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:15.118069 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:15.118186 (+ 117us) server_negotiation.cc:299] Negotiation successful
0806 01:02:15.118235 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":93,"threads_started":1}
I20260806 01:02:15.118323 10517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:15.114220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:40939 (local address 127.0.0.1:50560)
0806 01:02:15.114432 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:15.114441 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:15.114485 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:15.114880 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:15.114883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:15.114885 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:15.114971 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:15.114973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.116081 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.116083 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:15.117107 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:15.117110 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.117983 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.117984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:15.118062 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:15.118078 (+ 16us) client_negotiation.cc:769] Sending connection context
0806 01:02:15.118155 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 01:02:15.118213 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":36,"thread_start_us":68,"threads_started":1}
I20260806 01:02:15.119448 10166 tablet_service.cc:1467] Processing CreateTablet for tablet cb18cf4d93554805877e39626c7835d9 (DEFAULT_TABLE table=test-table [id=4a797ff8238e4489bdcd3dac3d923aa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:02:15.119802 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:15.114012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:36365 (local address 127.0.0.1:51902)
0806 01:02:15.114205 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:15.114251 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:15.114322 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:15.114901 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:15.114902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:15.114904 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:15.114973 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:15.114975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.116151 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.116152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:15.117806 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:15.117814 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.119322 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.119323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:15.119399 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:15.119403 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 01:02:15.119465 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 01:02:15.119718 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":5,"thread_start_us":60,"threads_started":1}
I20260806 01:02:15.119940 10166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb18cf4d93554805877e39626c7835d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:15.120168 10519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:15.114285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51902 (local address 127.23.242.2:36365)
0806 01:02:15.114720 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:15.114725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:15.114736 (+ 11us) server_negotiation.cc:407] Connection header received
0806 01:02:15.114777 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:15.114779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:15.114794 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:15.115177 (+ 383us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:15.115226 (+ 49us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.116052 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.117900 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.118797 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.118888 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:15.118976 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:15.120032 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0806 01:02:15.120088 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":110,"threads_started":1}
I20260806 01:02:15.120370 10516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:15.114012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:44319 (local address 127.0.0.1:38560)
0806 01:02:15.114214 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:15.114253 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:15.114320 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:15.116415 (+ 2095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:15.116416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:15.116419 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:15.116459 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:15.116462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.117750 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.117751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:15.118872 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:15.118875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.120030 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.120032 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:15.120159 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:15.120164 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 01:02:15.120255 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 01:02:15.120299 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":4,"thread_start_us":120,"threads_started":1}
I20260806 01:02:15.120544 10521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:15.114520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38560 (local address 127.23.242.3:44319)
0806 01:02:15.116240 (+ 1720us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:15.116244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:15.116254 (+ 10us) server_negotiation.cc:407] Connection header received
0806 01:02:15.116295 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:15.116297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:15.116313 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:15.116398 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:15.116547 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.117636 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.118969 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.119825 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.119904 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:15.119987 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:15.120339 (+ 352us) server_negotiation.cc:299] Negotiation successful
0806 01:02:15.120392 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1678,"thread_start_us":135,"threads_started":1}
I20260806 01:02:15.121047 10302 tablet_service.cc:1467] Processing CreateTablet for tablet cb18cf4d93554805877e39626c7835d9 (DEFAULT_TABLE table=test-table [id=4a797ff8238e4489bdcd3dac3d923aa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:02:15.121487 10302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb18cf4d93554805877e39626c7835d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:15.121501 10437 tablet_service.cc:1467] Processing CreateTablet for tablet cb18cf4d93554805877e39626c7835d9 (DEFAULT_TABLE table=test-table [id=4a797ff8238e4489bdcd3dac3d923aa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:02:15.121848 10437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb18cf4d93554805877e39626c7835d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:15.124955 10522 tablet_bootstrap.cc:492] T cb18cf4d93554805877e39626c7835d9 P 30f5789e0f8c456e8540bac21bff5d70: Bootstrap starting.
I20260806 01:02:15.126001 10522 tablet_bootstrap.cc:654] T cb18cf4d93554805877e39626c7835d9 P 30f5789e0f8c456e8540bac21bff5d70: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:15.125996 10524 tablet_bootstrap.cc:492] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417: Bootstrap starting.
I20260806 01:02:15.126127 10523 tablet_bootstrap.cc:492] T cb18cf4d93554805877e39626c7835d9 P 019f17402f4f4b7fa68fea4ed7ecee3c: Bootstrap starting.
I20260806 01:02:15.126529 10522 log.cc:824] T cb18cf4d93554805877e39626c7835d9 P 30f5789e0f8c456e8540bac21bff5d70: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:15.127287 10524 tablet_bootstrap.cc:654] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:15.127282 10523 tablet_bootstrap.cc:654] T cb18cf4d93554805877e39626c7835d9 P 019f17402f4f4b7fa68fea4ed7ecee3c: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:15.127748 10522 tablet_bootstrap.cc:492] T cb18cf4d93554805877e39626c7835d9 P 30f5789e0f8c456e8540bac21bff5d70: No bootstrap required, opened a new log
I20260806 01:02:15.127759 10524 log.cc:824] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:15.127786 10523 log.cc:824] T cb18cf4d93554805877e39626c7835d9 P 019f17402f4f4b7fa68fea4ed7ecee3c: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:15.127830 10522 ts_tablet_manager.cc:1397] T cb18cf4d93554805877e39626c7835d9 P 30f5789e0f8c456e8540bac21bff5d70: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 01:02:15.129191 10523 tablet_bootstrap.cc:492] T cb18cf4d93554805877e39626c7835d9 P 019f17402f4f4b7fa68fea4ed7ecee3c: No bootstrap required, opened a new log
I20260806 01:02:15.129194 10524 tablet_bootstrap.cc:492] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417: No bootstrap required, opened a new log
I20260806 01:02:15.129284 10523 ts_tablet_manager.cc:1397] T cb18cf4d93554805877e39626c7835d9 P 019f17402f4f4b7fa68fea4ed7ecee3c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260806 01:02:15.129284 10524 ts_tablet_manager.cc:1397] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 01:02:15.130687 10522 raft_consensus.cc:348] T cb18cf4d93554805877e39626c7835d9 P 30f5789e0f8c456e8540bac21bff5d70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019f17402f4f4b7fa68fea4ed7ecee3c" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36365 } } peers { permanent_uuid: "30f5789e0f8c456e8540bac21bff5d70" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 40939 } } peers { permanent_uuid: "13e46e15b59f4bdab76285152d690417" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 44319 } }
I20260806 01:02:15.130900 10522 raft_consensus.cc:374] T cb18cf4d93554805877e39626c7835d9 P 30f5789e0f8c456e8540bac21bff5d70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:15.130947 10522 raft_consensus.cc:729] T cb18cf4d93554805877e39626c7835d9 P 30f5789e0f8c456e8540bac21bff5d70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f5789e0f8c456e8540bac21bff5d70, State: Initialized, Role: FOLLOWER
I20260806 01:02:15.131093 10522 consensus_queue.cc:260] T cb18cf4d93554805877e39626c7835d9 P 30f5789e0f8c456e8540bac21bff5d70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019f17402f4f4b7fa68fea4ed7ecee3c" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36365 } } peers { permanent_uuid: "30f5789e0f8c456e8540bac21bff5d70" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 40939 } } peers { permanent_uuid: "13e46e15b59f4bdab76285152d690417" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 44319 } }
I20260806 01:02:15.131434 10522 ts_tablet_manager.cc:1428] T cb18cf4d93554805877e39626c7835d9 P 30f5789e0f8c456e8540bac21bff5d70: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260806 01:02:15.131474 10232 heartbeater.cc:503] Master 127.23.242.62:43919 was elected leader, sending a full tablet report...
I20260806 01:02:15.132302 10524 raft_consensus.cc:348] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019f17402f4f4b7fa68fea4ed7ecee3c" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36365 } } peers { permanent_uuid: "30f5789e0f8c456e8540bac21bff5d70" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 40939 } } peers { permanent_uuid: "13e46e15b59f4bdab76285152d690417" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 44319 } }
I20260806 01:02:15.132575 10524 raft_consensus.cc:374] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:15.132625 10524 raft_consensus.cc:729] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e46e15b59f4bdab76285152d690417, State: Initialized, Role: FOLLOWER
I20260806 01:02:15.132773 10524 consensus_queue.cc:260] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019f17402f4f4b7fa68fea4ed7ecee3c" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36365 } } peers { permanent_uuid: "30f5789e0f8c456e8540bac21bff5d70" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 40939 } } peers { permanent_uuid: "13e46e15b59f4bdab76285152d690417" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 44319 } }
I20260806 01:02:15.132731 10523 raft_consensus.cc:348] T cb18cf4d93554805877e39626c7835d9 P 019f17402f4f4b7fa68fea4ed7ecee3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019f17402f4f4b7fa68fea4ed7ecee3c" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36365 } } peers { permanent_uuid: "30f5789e0f8c456e8540bac21bff5d70" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 40939 } } peers { permanent_uuid: "13e46e15b59f4bdab76285152d690417" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 44319 } }
I20260806 01:02:15.132875 10523 raft_consensus.cc:374] T cb18cf4d93554805877e39626c7835d9 P 019f17402f4f4b7fa68fea4ed7ecee3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:15.132920 10523 raft_consensus.cc:729] T cb18cf4d93554805877e39626c7835d9 P 019f17402f4f4b7fa68fea4ed7ecee3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 019f17402f4f4b7fa68fea4ed7ecee3c, State: Initialized, Role: FOLLOWER
I20260806 01:02:15.133064 10524 ts_tablet_manager.cc:1428] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260806 01:02:15.133064 10523 consensus_queue.cc:260] T cb18cf4d93554805877e39626c7835d9 P 019f17402f4f4b7fa68fea4ed7ecee3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019f17402f4f4b7fa68fea4ed7ecee3c" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36365 } } peers { permanent_uuid: "30f5789e0f8c456e8540bac21bff5d70" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 40939 } } peers { permanent_uuid: "13e46e15b59f4bdab76285152d690417" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 44319 } }
I20260806 01:02:15.133319 10523 ts_tablet_manager.cc:1428] T cb18cf4d93554805877e39626c7835d9 P 019f17402f4f4b7fa68fea4ed7ecee3c: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260806 01:02:15.133315 10504 heartbeater.cc:503] Master 127.23.242.62:43919 was elected leader, sending a full tablet report...
I20260806 01:02:15.133457 10368 heartbeater.cc:503] Master 127.23.242.62:43919 was elected leader, sending a full tablet report...
W20260806 01:02:15.165256 10369 tablet.cc:2369] T cb18cf4d93554805877e39626c7835d9 P 019f17402f4f4b7fa68fea4ed7ecee3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 01:02:15.245134 10233 tablet.cc:2369] T cb18cf4d93554805877e39626c7835d9 P 30f5789e0f8c456e8540bac21bff5d70: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 01:02:15.338860 10505 tablet.cc:2369] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 01:02:15.343151 10529 raft_consensus.cc:482] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 01:02:15.343379 10529 raft_consensus.cc:504] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019f17402f4f4b7fa68fea4ed7ecee3c" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36365 } } peers { permanent_uuid: "30f5789e0f8c456e8540bac21bff5d70" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 40939 } } peers { permanent_uuid: "13e46e15b59f4bdab76285152d690417" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 44319 } }
I20260806 01:02:15.343945 10529 leader_election.cc:288] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 019f17402f4f4b7fa68fea4ed7ecee3c (127.23.242.2:36365), 30f5789e0f8c456e8540bac21bff5d70 (127.23.242.1:40939)
I20260806 01:02:15.348846 10506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:15.344255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:36365 (local address 127.23.242.3:38541)
0806 01:02:15.344490 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:15.344511 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:15.344599 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:15.344934 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:15.344936 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:15.344938 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:15.345026 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:15.345029 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.346005 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.346006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:15.347572 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:15.347575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.348524 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.348525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:15.348623 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:15.348627 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 01:02:15.348702 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 01:02:15.348754 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260806 01:02:15.348910 10520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:15.344603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:41281 (local address 127.23.242.1:40939)
0806 01:02:15.344708 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:15.344711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:15.344905 (+ 194us) server_negotiation.cc:407] Connection header received
0806 01:02:15.344940 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:15.344941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:15.344956 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:15.345058 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:15.345395 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.346343 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.347881 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.348517 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.348592 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:15.348599 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:15.348785 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 01:02:15.348836 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 01:02:15.348907 10519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:15.344489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:38541 (local address 127.23.242.2:36365)
0806 01:02:15.344562 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:15.344565 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:15.344606 (+ 41us) server_negotiation.cc:407] Connection header received
0806 01:02:15.344757 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:15.344759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:15.344779 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:15.344872 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:15.345216 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.345906 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.347703 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.348371 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.348491 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:15.348501 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:15.348775 (+ 274us) server_negotiation.cc:299] Negotiation successful
0806 01:02:15.348828 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 01:02:15.348852 10531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:15.344401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:40939 (local address 127.23.242.3:41281)
0806 01:02:15.344748 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:15.344764 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:15.344866 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:15.345185 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:15.345186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:15.345188 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:15.345263 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:15.345264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.346440 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.346441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:15.347790 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:15.347792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.348600 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.348601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:15.348686 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:15.348690 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 01:02:15.348752 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 01:02:15.348779 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":240,"threads_started":1}
I20260806 01:02:15.349411 10322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb18cf4d93554805877e39626c7835d9" candidate_uuid: "13e46e15b59f4bdab76285152d690417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "019f17402f4f4b7fa68fea4ed7ecee3c" is_pre_election: true
I20260806 01:02:15.349404 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb18cf4d93554805877e39626c7835d9" candidate_uuid: "13e46e15b59f4bdab76285152d690417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f5789e0f8c456e8540bac21bff5d70" is_pre_election: true
I20260806 01:02:15.349583 10322 raft_consensus.cc:2454] T cb18cf4d93554805877e39626c7835d9 P 019f17402f4f4b7fa68fea4ed7ecee3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e46e15b59f4bdab76285152d690417 in term 0.
I20260806 01:02:15.349602 10186 raft_consensus.cc:2454] T cb18cf4d93554805877e39626c7835d9 P 30f5789e0f8c456e8540bac21bff5d70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13e46e15b59f4bdab76285152d690417 in term 0.
I20260806 01:02:15.349861 10390 leader_election.cc:302] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13e46e15b59f4bdab76285152d690417, 30f5789e0f8c456e8540bac21bff5d70; no voters:
I20260806 01:02:15.350013 10529 raft_consensus.cc:2781] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 01:02:15.350077 10529 raft_consensus.cc:482] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 01:02:15.350106 10529 raft_consensus.cc:3037] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:15.351239 10529 raft_consensus.cc:504] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019f17402f4f4b7fa68fea4ed7ecee3c" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36365 } } peers { permanent_uuid: "30f5789e0f8c456e8540bac21bff5d70" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 40939 } } peers { permanent_uuid: "13e46e15b59f4bdab76285152d690417" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 44319 } }
I20260806 01:02:15.351442 10529 leader_election.cc:288] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417 [CANDIDATE]: Term 1 election: Requested vote from peers 019f17402f4f4b7fa68fea4ed7ecee3c (127.23.242.2:36365), 30f5789e0f8c456e8540bac21bff5d70 (127.23.242.1:40939)
I20260806 01:02:15.351723 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb18cf4d93554805877e39626c7835d9" candidate_uuid: "13e46e15b59f4bdab76285152d690417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f5789e0f8c456e8540bac21bff5d70"
I20260806 01:02:15.351773 10322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb18cf4d93554805877e39626c7835d9" candidate_uuid: "13e46e15b59f4bdab76285152d690417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "019f17402f4f4b7fa68fea4ed7ecee3c"
I20260806 01:02:15.351840 10186 raft_consensus.cc:3037] T cb18cf4d93554805877e39626c7835d9 P 30f5789e0f8c456e8540bac21bff5d70 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:15.351864 10322 raft_consensus.cc:3037] T cb18cf4d93554805877e39626c7835d9 P 019f17402f4f4b7fa68fea4ed7ecee3c [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:15.352661 10186 raft_consensus.cc:2454] T cb18cf4d93554805877e39626c7835d9 P 30f5789e0f8c456e8540bac21bff5d70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e46e15b59f4bdab76285152d690417 in term 1.
I20260806 01:02:15.352670 10322 raft_consensus.cc:2454] T cb18cf4d93554805877e39626c7835d9 P 019f17402f4f4b7fa68fea4ed7ecee3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13e46e15b59f4bdab76285152d690417 in term 1.
I20260806 01:02:15.352856 10390 leader_election.cc:302] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 019f17402f4f4b7fa68fea4ed7ecee3c, 13e46e15b59f4bdab76285152d690417; no voters:
I20260806 01:02:15.352998 10529 raft_consensus.cc:2781] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:15.353214 10529 raft_consensus.cc:686] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417 [term 1 LEADER]: Becoming Leader. State: Replica: 13e46e15b59f4bdab76285152d690417, State: Running, Role: LEADER
I20260806 01:02:15.353327 10529 consensus_queue.cc:237] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019f17402f4f4b7fa68fea4ed7ecee3c" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36365 } } peers { permanent_uuid: "30f5789e0f8c456e8540bac21bff5d70" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 40939 } } peers { permanent_uuid: "13e46e15b59f4bdab76285152d690417" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 44319 } }
I20260806 01:02:15.354267 10037 catalog_manager.cc:5495] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13e46e15b59f4bdab76285152d690417 (127.23.242.3). New cstate: current_term: 1 leader_uuid: "13e46e15b59f4bdab76285152d690417" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019f17402f4f4b7fa68fea4ed7ecee3c" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30f5789e0f8c456e8540bac21bff5d70" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 40939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13e46e15b59f4bdab76285152d690417" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 44319 } health_report { overall_health: HEALTHY } } }
I20260806 01:02:15.409626 10521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:15.405648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38574 (local address 127.23.242.3:44319)
0806 01:02:15.405752 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:15.405754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:15.405767 (+ 13us) server_negotiation.cc:407] Connection header received
0806 01:02:15.405827 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:15.405828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:15.405863 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:15.405945 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 01:02:15.406163 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.407166 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.408413 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:15.408682 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:15.408767 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:15.409037 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 01:02:15.409160 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 01:02:15.409243 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:15.409447 (+ 204us) server_negotiation.cc:299] Negotiation successful
0806 01:02:15.409480 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 01:02:15.412266 10186 raft_consensus.cc:1261] T cb18cf4d93554805877e39626c7835d9 P 30f5789e0f8c456e8540bac21bff5d70 [term 1 FOLLOWER]: Refusing update from remote peer 13e46e15b59f4bdab76285152d690417: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:02:15.412266 10322 raft_consensus.cc:1261] T cb18cf4d93554805877e39626c7835d9 P 019f17402f4f4b7fa68fea4ed7ecee3c [term 1 FOLLOWER]: Refusing update from remote peer 13e46e15b59f4bdab76285152d690417: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:02:15.412708 10529 consensus_queue.cc:1035] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "019f17402f4f4b7fa68fea4ed7ecee3c" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 36365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:02:15.412897 10532 consensus_queue.cc:1035] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30f5789e0f8c456e8540bac21bff5d70" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 40939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:02:15.415688 10536 mvcc.cc:205] Tried to move back new op lower bound from 7315366442644123648 to 7315366442407886848. Current Snapshot: MvccSnapshot[applied={T|T < 7315366442644123648}]
I20260806 01:02:15.415900 10537 mvcc.cc:205] Tried to move back new op lower bound from 7315366442644123648 to 7315366442407886848. Current Snapshot: MvccSnapshot[applied={T|T < 7315366442644123648}]
I20260806 01:02:15.416487 10538 mvcc.cc:205] Tried to move back new op lower bound from 7315366442644123648 to 7315366442407886848. Current Snapshot: MvccSnapshot[applied={T|T < 7315366442644123648}]
I20260806 01:02:15.418809 10036 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 01:02:15.418924 10036 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 01:02:15.420260 10036 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a77d5ca37274ec0b021de23af913f28: Sending DeleteTablet for 3 replicas of tablet cb18cf4d93554805877e39626c7835d9
I20260806 01:02:15.420711 10302 tablet_service.cc:1514] Processing DeleteTablet for tablet cb18cf4d93554805877e39626c7835d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:02:15 UTC) from {username='slave'} at 127.0.0.1:51902
I20260806 01:02:15.420773 10437 tablet_service.cc:1514] Processing DeleteTablet for tablet cb18cf4d93554805877e39626c7835d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:02:15 UTC) from {username='slave'} at 127.0.0.1:38560
I20260806 01:02:15.420750 10166 tablet_service.cc:1514] Processing DeleteTablet for tablet cb18cf4d93554805877e39626c7835d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:02:15 UTC) from {username='slave'} at 127.0.0.1:50560
I20260806 01:02:15.421123 10546 tablet_replica.cc:330] T cb18cf4d93554805877e39626c7835d9 P 019f17402f4f4b7fa68fea4ed7ecee3c: stopping tablet replica
I20260806 01:02:15.421553 10546 raft_consensus.cc:2229] T cb18cf4d93554805877e39626c7835d9 P 019f17402f4f4b7fa68fea4ed7ecee3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 01:02:15.421924 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 10100
I20260806 01:02:15.421949 10546 raft_consensus.cc:2258] T cb18cf4d93554805877e39626c7835d9 P 019f17402f4f4b7fa68fea4ed7ecee3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 01:02:15.423336 10546 ts_tablet_manager.cc:1905] T cb18cf4d93554805877e39626c7835d9 P 019f17402f4f4b7fa68fea4ed7ecee3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 01:02:15.426785 10548 tablet_replica.cc:330] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417: stopping tablet replica
I20260806 01:02:15.426757 10546 ts_tablet_manager.cc:1918] T cb18cf4d93554805877e39626c7835d9 P 019f17402f4f4b7fa68fea4ed7ecee3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 01:02:15.426949 10548 raft_consensus.cc:2229] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417 [term 1 LEADER]: Raft consensus shutting down.
I20260806 01:02:15.427039 10546 log.cc:1196] T cb18cf4d93554805877e39626c7835d9 P 019f17402f4f4b7fa68fea4ed7ecee3c: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-1/wal/wals/cb18cf4d93554805877e39626c7835d9
I20260806 01:02:15.427117 10548 raft_consensus.cc:2258] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 01:02:15.427652 10546 ts_tablet_manager.cc:1939] T cb18cf4d93554805877e39626c7835d9 P 019f17402f4f4b7fa68fea4ed7ecee3c: Deleting consensus metadata
I20260806 01:02:15.428354 10023 catalog_manager.cc:4841] TS 019f17402f4f4b7fa68fea4ed7ecee3c (127.23.242.2:36365): tablet cb18cf4d93554805877e39626c7835d9 (table test-table [id=4a797ff8238e4489bdcd3dac3d923aa6]) successfully deleted
W20260806 01:02:15.432075 10023 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.242.1:40939 (error 108)
I20260806 01:02:15.432366 10548 ts_tablet_manager.cc:1905] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 01:02:15.432569 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 10235
I20260806 01:02:15.436492 10548 ts_tablet_manager.cc:1918] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 01:02:15.436623 10548 log.cc:1196] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785977869987768-24520-0/minicluster-data/ts-2/wal/wals/cb18cf4d93554805877e39626c7835d9
I20260806 01:02:15.436894 10548 ts_tablet_manager.cc:1939] T cb18cf4d93554805877e39626c7835d9 P 13e46e15b59f4bdab76285152d690417: Deleting consensus metadata
I20260806 01:02:15.437389 10022 catalog_manager.cc:4841] TS 13e46e15b59f4bdab76285152d690417 (127.23.242.3:44319): tablet cb18cf4d93554805877e39626c7835d9 (table test-table [id=4a797ff8238e4489bdcd3dac3d923aa6]) successfully deleted
I20260806 01:02:15.437475 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:15.436405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:40939 (local address 127.0.0.1:50568)
0806 01:02:15.437325 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:15.437391 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.1:40939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":891}
W20260806 01:02:15.437608 10023 catalog_manager.cc:4584] TS 30f5789e0f8c456e8540bac21bff5d70 (127.23.242.1:40939): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cb18cf4d93554805877e39626c7835d9: Network error: Client connection negotiation failed: client connection to 127.23.242.1:40939: connect: Connection refused (error 111)
I20260806 01:02:15.443281 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 10371
I20260806 01:02:15.451751 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 10005
2026-08-06T01:02:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1101 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 01:02:15 dist-test-slave-dhph krb5kdc[10552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 01:02:15 dist-test-slave-dhph krb5kdc[10552](info): set up 2 sockets
Aug 06 01:02:15 dist-test-slave-dhph krb5kdc[10552](info): commencing operation
krb5kdc: starting...
W20260806 01:02:16.784931 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.310s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 01:02:16 dist-test-slave-dhph krb5kdc[10552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T01:02:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:02:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 01:02:16.949396 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:45707
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:45707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:17.050874 10566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:17.051206 10566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:17.051270 10566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:17.053241 10566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:17.053332 10566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:17.053364 10566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:17.053403 10566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:17.053432 10566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:17.055459 10566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:45707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:45707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:17.055821 10566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:17.056104 10566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:17.063663 10574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:17.063586 10572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:17.063624 10571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:17.063843 10566 server_base.cc:1034] running on GCE node
I20260806 01:02:17.064554 10566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:17.064924 10566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:17.066124 10566 hybrid_clock.cc:648] HybridClock initialized: now 1785978137066081 us; error 62 us; skew 500 ppm
Aug 06 01:02:17 dist-test-slave-dhph krb5kdc[10552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978137, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:17.069546 10566 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 01:02:17.070361 10566 webserver.cc:460] Webserver started at http://127.23.242.62:41289/ using document root <none> and password file <none>
I20260806 01:02:17.070822 10566 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:17.070927 10566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:17.071111 10566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:17.072445 10566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "a00713e064f142b883632e34251a1271"
format_stamp: "Formatted at 2026-08-06 01:02:17 on dist-test-slave-dhph"
server_key: "7ca594d94cce01e3a0f6f56aa68b00b6"
server_key_iv: "a42250b7c82ed26d8b7f6e9467ee9b78"
server_key_version: "encryptionkey@0"
I20260806 01:02:17.072888 10566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "a00713e064f142b883632e34251a1271"
format_stamp: "Formatted at 2026-08-06 01:02:17 on dist-test-slave-dhph"
server_key: "7ca594d94cce01e3a0f6f56aa68b00b6"
server_key_iv: "a42250b7c82ed26d8b7f6e9467ee9b78"
server_key_version: "encryptionkey@0"
I20260806 01:02:17.074723 10566 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260806 01:02:17.076001 10581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:17.076375 10566 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:17.076524 10566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "a00713e064f142b883632e34251a1271"
format_stamp: "Formatted at 2026-08-06 01:02:17 on dist-test-slave-dhph"
server_key: "7ca594d94cce01e3a0f6f56aa68b00b6"
server_key_iv: "a42250b7c82ed26d8b7f6e9467ee9b78"
server_key_version: "encryptionkey@0"
I20260806 01:02:17.076615 10566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:17.088603 10566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:17.091595 10566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:17.091845 10566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:17.097258 10566 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:45707
I20260806 01:02:17.097301 10633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:45707 every 8 connection(s)
I20260806 01:02:17.097730 10566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:02:17.098785 10634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:17.101227 10634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a00713e064f142b883632e34251a1271: Bootstrap starting.
I20260806 01:02:17.101856 10634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a00713e064f142b883632e34251a1271: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:17.102178 10634 log.cc:824] T 00000000000000000000000000000000 P a00713e064f142b883632e34251a1271: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:17.103461 10634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a00713e064f142b883632e34251a1271: No bootstrap required, opened a new log
I20260806 01:02:17.105644 10634 raft_consensus.cc:348] T 00000000000000000000000000000000 P a00713e064f142b883632e34251a1271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00713e064f142b883632e34251a1271" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 45707 } }
I20260806 01:02:17.105896 10634 raft_consensus.cc:374] T 00000000000000000000000000000000 P a00713e064f142b883632e34251a1271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:17.105922 10634 raft_consensus.cc:729] T 00000000000000000000000000000000 P a00713e064f142b883632e34251a1271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a00713e064f142b883632e34251a1271, State: Initialized, Role: FOLLOWER
I20260806 01:02:17.106127 10634 consensus_queue.cc:260] T 00000000000000000000000000000000 P a00713e064f142b883632e34251a1271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00713e064f142b883632e34251a1271" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 45707 } }
I20260806 01:02:17.106222 10634 raft_consensus.cc:388] T 00000000000000000000000000000000 P a00713e064f142b883632e34251a1271 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:02:17.106249 10634 raft_consensus.cc:482] T 00000000000000000000000000000000 P a00713e064f142b883632e34251a1271 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:02:17.106351 10634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a00713e064f142b883632e34251a1271 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:17.107146 10634 raft_consensus.cc:504] T 00000000000000000000000000000000 P a00713e064f142b883632e34251a1271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00713e064f142b883632e34251a1271" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 45707 } }
I20260806 01:02:17.107249 10634 leader_election.cc:302] T 00000000000000000000000000000000 P a00713e064f142b883632e34251a1271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a00713e064f142b883632e34251a1271; no voters:
I20260806 01:02:17.107442 10634 leader_election.cc:288] T 00000000000000000000000000000000 P a00713e064f142b883632e34251a1271 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:02:17.107474 10637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a00713e064f142b883632e34251a1271 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:17.107617 10637 raft_consensus.cc:686] T 00000000000000000000000000000000 P a00713e064f142b883632e34251a1271 [term 1 LEADER]: Becoming Leader. State: Replica: a00713e064f142b883632e34251a1271, State: Running, Role: LEADER
I20260806 01:02:17.107885 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 10566
I20260806 01:02:17.108016 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:02:17.108248 24520 external_mini_cluster.cc:1428] Setting key 568fbef366e42bc98adcdf408ca12a9c
I20260806 01:02:17.107856 10634 sys_catalog.cc:564] T 00000000000000000000000000000000 P a00713e064f142b883632e34251a1271 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:17.108476 10637 consensus_queue.cc:237] T 00000000000000000000000000000000 P a00713e064f142b883632e34251a1271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00713e064f142b883632e34251a1271" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 45707 } }
I20260806 01:02:17.109982 10639 sys_catalog.cc:455] T 00000000000000000000000000000000 P a00713e064f142b883632e34251a1271 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a00713e064f142b883632e34251a1271" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00713e064f142b883632e34251a1271" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 45707 } } }
I20260806 01:02:17.110122 10639 sys_catalog.cc:458] T 00000000000000000000000000000000 P a00713e064f142b883632e34251a1271 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:17.110378 10638 sys_catalog.cc:455] T 00000000000000000000000000000000 P a00713e064f142b883632e34251a1271 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a00713e064f142b883632e34251a1271. Latest consensus state: current_term: 1 leader_uuid: "a00713e064f142b883632e34251a1271" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00713e064f142b883632e34251a1271" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 45707 } } }
I20260806 01:02:17.110473 10638 sys_catalog.cc:458] T 00000000000000000000000000000000 P a00713e064f142b883632e34251a1271 [sys.catalog]: This master's current role is: LEADER
W20260806 01:02:17.111358 10656 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a00713e064f142b883632e34251a1271: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 01:02:17.111423 10656 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 01:02:17.111506 10649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:02:17.111740 10649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:02:17.113682 10649 catalog_manager.cc:1347] Generated new cluster ID: 0f4e4c242c2344c588be46adebdaf120
I20260806 01:02:17.113754 10649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 01:02:17 dist-test-slave-dhph krb5kdc[10552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:02:17.121521 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:17.108855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32926 (local address 127.23.242.62:45707)
0806 01:02:17.109363 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:17.109378 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:17.109424 (+ 46us) server_negotiation.cc:407] Connection header received
0806 01:02:17.109585 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:17.109593 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:17.109661 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:17.110759 (+ 1098us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:17.111113 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.112551 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.113621 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.114083 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.114220 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:17.116885 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:17.116889 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:17.116890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:17.116903 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:17.119738 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:17.120195 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:17.120199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:17.120200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:17.120320 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:17.120569 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:17.120570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:17.120570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:17.120852 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:17.121058 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:17.121135 (+ 77us) server_negotiation.cc:299] Negotiation successful
0806 01:02:17.121312 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":161,"threads_started":1}
I20260806 01:02:17.168742 10649 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:02:17.169433 10649 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:02:17.177568 10649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a00713e064f142b883632e34251a1271: Generated new TSK 0
I20260806 01:02:17.177831 10649 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.242.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
I20260806 01:02:17.228842 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:0
--local_ip_for_outbound_sockets=127.23.242.1
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:45707
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:17.328392 10661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:17.328660 10661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:17.328692 10661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:17.329975 10661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:17.330041 10661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:17.330075 10661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 01:02:17.331856 10661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:45707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:17.332162 10661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:17.332453 10661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:17.340873 10670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:17.340889 10667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:17.340906 10668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:17.340911 10661 server_base.cc:1034] running on GCE node
I20260806 01:02:17.341526 10661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:17.341886 10661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:17.343066 10661 hybrid_clock.cc:648] HybridClock initialized: now 1785978137343036 us; error 50 us; skew 500 ppm
Aug 06 01:02:17 dist-test-slave-dhph krb5kdc[10552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978137, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:17.346542 10661 init.cc:377] Logged in from keytab as kudu/127.23.242.1@KRBTEST.COM (short username kudu)
I20260806 01:02:17.347077 10661 webserver.cc:460] Webserver started at http://127.23.242.1:37331/ using document root <none> and password file <none>
I20260806 01:02:17.347358 10661 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:17.347429 10661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:17.347585 10661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:17.348871 10661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-0/data/instance:
uuid: "ec5b0efaf235436b81a28cd4320c3a7a"
format_stamp: "Formatted at 2026-08-06 01:02:17 on dist-test-slave-dhph"
server_key: "00448e7efe5689d8c71d051ef66de09f"
server_key_iv: "3b6fe6a3e8c9f972563f86a8bbab2d14"
server_key_version: "encryptionkey@0"
I20260806 01:02:17.349448 10661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance:
uuid: "ec5b0efaf235436b81a28cd4320c3a7a"
format_stamp: "Formatted at 2026-08-06 01:02:17 on dist-test-slave-dhph"
server_key: "00448e7efe5689d8c71d051ef66de09f"
server_key_iv: "3b6fe6a3e8c9f972563f86a8bbab2d14"
server_key_version: "encryptionkey@0"
I20260806 01:02:17.351136 10661 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260806 01:02:17.352173 10677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:17.352517 10661 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:17.352660 10661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "ec5b0efaf235436b81a28cd4320c3a7a"
format_stamp: "Formatted at 2026-08-06 01:02:17 on dist-test-slave-dhph"
server_key: "00448e7efe5689d8c71d051ef66de09f"
server_key_iv: "3b6fe6a3e8c9f972563f86a8bbab2d14"
server_key_version: "encryptionkey@0"
I20260806 01:02:17.352753 10661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:17.367307 10661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:17.371281 10661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:17.371803 10661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:17.372402 10661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:02:17.372967 10661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:02:17.373040 10661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:17.373124 10661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:02:17.373176 10661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:17.380812 10661 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:38291
I20260806 01:02:17.380894 10797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:38291 every 8 connection(s)
I20260806 01:02:17.381265 10661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 01:02:17.387306 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 10661
I20260806 01:02:17.387468 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance
I20260806 01:02:17.387681 24520 external_mini_cluster.cc:1428] Setting key 2a6ea454d47ca3f2ed372f34dc47cab5
Aug 06 01:02:17 dist-test-slave-dhph krb5kdc[10552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978137, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:02:17.394068 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:17.382696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:35507 (local address 127.23.242.62:45707)
0806 01:02:17.382787 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:17.382789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:17.382991 (+ 202us) server_negotiation.cc:407] Connection header received
0806 01:02:17.383467 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:17.383469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:17.383485 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:17.383566 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:17.384560 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.385300 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.386674 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.386894 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.386964 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:17.390720 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:17.390722 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:17.390722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:17.390733 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:17.392399 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:17.392933 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:17.392935 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:17.392935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:17.393004 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:17.393306 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:17.393307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:17.393307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:17.393442 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:17.393530 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:17.393858 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 01:02:17.393959 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 01:02:17.394361 10800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:17.382623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45707 (local address 127.23.242.1:35507)
0806 01:02:17.382807 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:17.382851 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:17.383339 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:17.383656 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:17.383659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:17.384245 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:17.384446 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:17.384451 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.385499 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.385500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:17.386528 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:17.386531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.387114 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.387117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:17.387286 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:17.388199 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:02:17.388214 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:17.390564 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:17.392534 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:17.392545 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:17.392547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:17.392836 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:17.393087 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:17.393088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:17.393089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:17.393223 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:17.393520 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:17.393529 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:17.393698 (+ 169us) client_negotiation.cc:769] Sending connection context
0806 01:02:17.393818 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 01:02:17.394020 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":106,"threads_started":1}
I20260806 01:02:17.395037 10798 heartbeater.cc:347] Connected to a master server at 127.23.242.62:45707
I20260806 01:02:17.395169 10798 heartbeater.cc:464] Registering TS with master...
I20260806 01:02:17.395413 10798 heartbeater.cc:511] Master 127.23.242.62:45707 requested a full tablet report, sending...
I20260806 01:02:17.396127 10597 ts_manager.cc:194] Registered new tserver with Master: ec5b0efaf235436b81a28cd4320c3a7a (127.23.242.1:38291)
I20260806 01:02:17.397189 10597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.1@KRBTEST.COM'} at 127.23.242.1:35507
WARNING: no policy specified for kudu/127.23.242.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
I20260806 01:02:17.426388 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:0
--local_ip_for_outbound_sockets=127.23.242.2
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:45707
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:17.521827 10803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:17.522192 10803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:17.522250 10803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:17.523681 10803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:17.523742 10803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:17.523774 10803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 01:02:17.525334 10803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:45707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:17.525569 10803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:17.525768 10803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:17.532407 10810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:17.532828 10812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:17.533080 10809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:17.533015 10803 server_base.cc:1034] running on GCE node
I20260806 01:02:17.533315 10803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:17.534253 10803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:17.535553 10803 hybrid_clock.cc:648] HybridClock initialized: now 1785978137535516 us; error 50 us; skew 500 ppm
Aug 06 01:02:17 dist-test-slave-dhph krb5kdc[10552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978137, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:17.539561 10803 init.cc:377] Logged in from keytab as kudu/127.23.242.2@KRBTEST.COM (short username kudu)
I20260806 01:02:17.540117 10803 webserver.cc:460] Webserver started at http://127.23.242.2:37197/ using document root <none> and password file <none>
I20260806 01:02:17.540416 10803 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:17.540470 10803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:17.540973 10803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:17.543018 10803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-1/data/instance:
uuid: "dc32bda386c34f228097d138403f8024"
format_stamp: "Formatted at 2026-08-06 01:02:17 on dist-test-slave-dhph"
server_key: "dce908e9bd8fa366981d04041d9edcb0"
server_key_iv: "e98cd5abdb062545ffc6e1d14979f49b"
server_key_version: "encryptionkey@0"
I20260806 01:02:17.543464 10803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance:
uuid: "dc32bda386c34f228097d138403f8024"
format_stamp: "Formatted at 2026-08-06 01:02:17 on dist-test-slave-dhph"
server_key: "dce908e9bd8fa366981d04041d9edcb0"
server_key_iv: "e98cd5abdb062545ffc6e1d14979f49b"
server_key_version: "encryptionkey@0"
I20260806 01:02:17.545150 10803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 01:02:17.546347 10819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:17.546764 10803 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:17.546911 10803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "dc32bda386c34f228097d138403f8024"
format_stamp: "Formatted at 2026-08-06 01:02:17 on dist-test-slave-dhph"
server_key: "dce908e9bd8fa366981d04041d9edcb0"
server_key_iv: "e98cd5abdb062545ffc6e1d14979f49b"
server_key_version: "encryptionkey@0"
I20260806 01:02:17.547008 10803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:17.575722 10803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:17.578362 10803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:17.578598 10803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:17.578931 10803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:02:17.579360 10803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:02:17.579424 10803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:17.579468 10803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:02:17.579510 10803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:17.587158 10803 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:41719
I20260806 01:02:17.587204 10935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:41719 every 8 connection(s)
I20260806 01:02:17.587698 10803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 01:02:17.594403 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 10803
I20260806 01:02:17.594620 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance
I20260806 01:02:17.594918 24520 external_mini_cluster.cc:1428] Setting key f6c322c397a5894cb2372e2e37b4f69a
Aug 06 01:02:17 dist-test-slave-dhph krb5kdc[10552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978137, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:02:17.603340 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:17.589080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:52183 (local address 127.23.242.62:45707)
0806 01:02:17.589137 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:17.589139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:17.589206 (+ 67us) server_negotiation.cc:407] Connection header received
0806 01:02:17.589807 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:17.589808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:17.589819 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:17.589890 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:17.591373 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.591942 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.593253 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.593534 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.593644 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:17.598729 (+ 5085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:17.598731 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:17.598732 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:17.598741 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:17.601401 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:17.601971 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:17.601972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:17.601973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:17.602043 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:17.602351 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:17.602352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:17.602353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:17.602500 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:17.602591 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:17.602983 (+ 392us) server_negotiation.cc:299] Negotiation successful
0806 01:02:17.603089 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260806 01:02:17.603621 10939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:17.588893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45707 (local address 127.23.242.2:52183)
0806 01:02:17.589073 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:17.589123 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:17.589666 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:17.590058 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:17.590062 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:17.590906 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:17.591147 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:17.591153 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.592097 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.592098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:17.593066 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:17.593069 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.593703 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.593704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:17.593846 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:17.594542 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:02:17.594556 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:17.598560 (+ 4004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:17.601540 (+ 2980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:17.601553 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:17.601556 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:17.601860 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:17.602122 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:17.602123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:17.602124 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:17.602267 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:17.602606 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:17.602616 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:17.602792 (+ 176us) client_negotiation.cc:769] Sending connection context
0806 01:02:17.602956 (+ 164us) client_negotiation.cc:240] Negotiation successful
0806 01:02:17.603158 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":101,"threads_started":1}
I20260806 01:02:17.604117 10936 heartbeater.cc:347] Connected to a master server at 127.23.242.62:45707
I20260806 01:02:17.604218 10936 heartbeater.cc:464] Registering TS with master...
I20260806 01:02:17.604449 10936 heartbeater.cc:511] Master 127.23.242.62:45707 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.242.3@KRBTEST.COM; defaulting to no policy
I20260806 01:02:17.605152 10597 ts_manager.cc:194] Registered new tserver with Master: dc32bda386c34f228097d138403f8024 (127.23.242.2:41719)
I20260806 01:02:17.605583 10597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.2@KRBTEST.COM'} at 127.23.242.2:52183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
I20260806 01:02:17.632575 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:0
--local_ip_for_outbound_sockets=127.23.242.3
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:45707
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:17.736594 10943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:17.736912 10943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:17.736949 10943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:17.739236 10943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:17.739467 10943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:17.739545 10943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 01:02:17.741855 10943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:45707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:17.742112 10943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:17.742327 10943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:17.748428 10951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:17.748441 10948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:17.748456 10949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:17.748486 10943 server_base.cc:1034] running on GCE node
I20260806 01:02:17.748912 10943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:17.749191 10943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:17.750523 10943 hybrid_clock.cc:648] HybridClock initialized: now 1785978137750476 us; error 49 us; skew 500 ppm
Aug 06 01:02:17 dist-test-slave-dhph krb5kdc[10552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978137, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:17.754153 10943 init.cc:377] Logged in from keytab as kudu/127.23.242.3@KRBTEST.COM (short username kudu)
I20260806 01:02:17.754868 10943 webserver.cc:460] Webserver started at http://127.23.242.3:42665/ using document root <none> and password file <none>
I20260806 01:02:17.755131 10943 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:17.755208 10943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:17.755371 10943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:17.756608 10943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-2/data/instance:
uuid: "b39b7491cd424f3c9dd933dba02b93d8"
format_stamp: "Formatted at 2026-08-06 01:02:17 on dist-test-slave-dhph"
server_key: "96b20764dabac4e8b4c0d7f44803c594"
server_key_iv: "b63def125e916361931359d1b406d9da"
server_key_version: "encryptionkey@0"
I20260806 01:02:17.757037 10943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance:
uuid: "b39b7491cd424f3c9dd933dba02b93d8"
format_stamp: "Formatted at 2026-08-06 01:02:17 on dist-test-slave-dhph"
server_key: "96b20764dabac4e8b4c0d7f44803c594"
server_key_iv: "b63def125e916361931359d1b406d9da"
server_key_version: "encryptionkey@0"
I20260806 01:02:17.758548 10943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 01:02:17.759680 10958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:17.760116 10943 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:17.760267 10943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "b39b7491cd424f3c9dd933dba02b93d8"
format_stamp: "Formatted at 2026-08-06 01:02:17 on dist-test-slave-dhph"
server_key: "96b20764dabac4e8b4c0d7f44803c594"
server_key_iv: "b63def125e916361931359d1b406d9da"
server_key_version: "encryptionkey@0"
I20260806 01:02:17.760371 10943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:17.774191 10943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:17.777633 10943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:17.777864 10943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:17.778169 10943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:02:17.778546 10943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:02:17.778645 10943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:17.778712 10943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:02:17.778748 10943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:17.785516 10943 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:41411
I20260806 01:02:17.785568 11079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:41411 every 8 connection(s)
I20260806 01:02:17.785931 10943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
I20260806 01:02:17.790668 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 10943
I20260806 01:02:17.790992 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance
I20260806 01:02:17.791344 24520 external_mini_cluster.cc:1428] Setting key bc982d4ef090eec29eeafdde6229efbe
Aug 06 01:02:17 dist-test-slave-dhph krb5kdc[10552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978137, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:02:17.799156 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:17.786818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:53241 (local address 127.23.242.62:45707)
0806 01:02:17.786878 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:17.786880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:17.787429 (+ 549us) server_negotiation.cc:407] Connection header received
0806 01:02:17.788181 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:17.788182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:17.788195 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:17.788273 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:17.789653 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.790455 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.792410 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.792628 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.792725 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:17.795550 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:17.795554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:17.795555 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:17.795569 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:17.797169 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:17.797685 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:17.797686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:17.797687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:17.797756 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:17.798341 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:17.798342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:17.798342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:17.798481 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:17.798563 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:17.798957 (+ 394us) server_negotiation.cc:299] Negotiation successful
0806 01:02:17.799046 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 01:02:17.799392 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:17.786836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:45707 (local address 127.23.242.3:53241)
0806 01:02:17.787281 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:17.787327 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:17.787911 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:17.788395 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:17.788399 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:17.789303 (+ 904us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:17.789518 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:17.789523 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.790654 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.790655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:17.792120 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:17.792123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.792735 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.792736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:17.792879 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:17.793457 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:02:17.793466 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:17.795413 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:17.797303 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:17.797315 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:17.797317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:17.797594 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:17.797839 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:17.797840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:17.797840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:17.798136 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:17.798567 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:17.798576 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:17.798795 (+ 219us) client_negotiation.cc:769] Sending connection context
0806 01:02:17.798923 (+ 128us) client_negotiation.cc:240] Negotiation successful
0806 01:02:17.799107 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":319,"threads_started":1}
I20260806 01:02:17.800099 11080 heartbeater.cc:347] Connected to a master server at 127.23.242.62:45707
I20260806 01:02:17.800388 11080 heartbeater.cc:464] Registering TS with master...
I20260806 01:02:17.800662 11080 heartbeater.cc:511] Master 127.23.242.62:45707 requested a full tablet report, sending...
I20260806 01:02:17.801213 10597 ts_manager.cc:194] Registered new tserver with Master: b39b7491cd424f3c9dd933dba02b93d8 (127.23.242.3:41411)
I20260806 01:02:17.801720 10597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.3@KRBTEST.COM'} at 127.23.242.3:53241
I20260806 01:02:17.804262 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 01:02:17.813299 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:17.805909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60274 (local address 127.23.242.62:45707)
0806 01:02:17.805982 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:17.805984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:17.805994 (+ 10us) server_negotiation.cc:407] Connection header received
0806 01:02:17.806067 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:17.806068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:17.806079 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:17.806168 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:17.806843 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.807348 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.808100 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.808376 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.808451 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:17.810000 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:17.810001 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:17.810002 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:17.810009 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:17.811813 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:17.812302 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:17.812304 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:17.812304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:17.812374 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:17.812629 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:17.812630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:17.812631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:17.812796 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:17.812923 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:17.813094 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 01:02:17.813164 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 01:02:17.815860 10597 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 01:02:17.817673 10597 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 01:02:17.825959 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:17.821469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:41719 (local address 127.0.0.1:57642)
0806 01:02:17.821969 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:17.822003 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:17.822142 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:17.822432 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:17.822440 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:17.822443 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:17.822546 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:17.822548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.823544 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.823545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:17.824619 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:17.824622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.825601 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.825602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:17.825707 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:17.825725 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 01:02:17.825798 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 01:02:17.825850 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":1,"spinlock_wait_cycles":896,"thread_start_us":94,"threads_started":1}
I20260806 01:02:17.826207 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:17.821780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57642 (local address 127.23.242.2:41719)
0806 01:02:17.822011 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:17.822017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:17.822092 (+ 75us) server_negotiation.cc:407] Connection header received
0806 01:02:17.822251 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:17.822254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:17.822273 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:17.822346 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:17.822660 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.823377 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.824725 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.825495 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.825572 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:17.825631 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:17.825827 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 01:02:17.825870 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":87,"threads_started":1}
I20260806 01:02:17.826974 10869 tablet_service.cc:1467] Processing CreateTablet for tablet bb0c8bfac735494187a8591d284df018 (DEFAULT_TABLE table=test-table [id=1e73ab20a19a4ff9bf6d1642121ef0c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:02:17.827137 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:17.821773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56642 (local address 127.23.242.3:41411)
0806 01:02:17.822106 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:17.822156 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:17.822205 (+ 49us) server_negotiation.cc:407] Connection header received
0806 01:02:17.822275 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:17.822278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:17.822301 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:17.822401 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:17.823681 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.824762 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.825993 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.826774 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.826850 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:17.826901 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:17.827024 (+ 123us) server_negotiation.cc:299] Negotiation successful
0806 01:02:17.827064 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":89,"threads_started":1}
I20260806 01:02:17.827147 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:17.821470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:41411 (local address 127.0.0.1:56642)
0806 01:02:17.822134 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:17.822145 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:17.822207 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:17.822485 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:17.822486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:17.822489 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:17.822547 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:17.822549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.824859 (+ 2310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.824860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:17.825898 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:17.825901 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.826872 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.826873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:17.826940 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:17.826945 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 01:02:17.827002 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 01:02:17.827046 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":5,"thread_start_us":79,"threads_started":1}
I20260806 01:02:17.827425 10869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb0c8bfac735494187a8591d284df018. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:17.827418 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:17.821936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41526 (local address 127.23.242.1:38291)
0806 01:02:17.822135 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:17.822139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:17.822352 (+ 213us) server_negotiation.cc:407] Connection header received
0806 01:02:17.822398 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:17.822400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:17.822415 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:17.822492 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:17.823483 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.824646 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.825697 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.826861 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.826930 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:17.826987 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:17.827268 (+ 281us) server_negotiation.cc:299] Negotiation successful
0806 01:02:17.827316 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":99,"threads_started":1}
I20260806 01:02:17.827626 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:17.821672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:38291 (local address 127.0.0.1:41526)
0806 01:02:17.822259 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:17.822269 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:17.822321 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:17.822501 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:17.822501 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:17.822503 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:17.822546 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:17.822548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.824770 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.824771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:17.825589 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:17.825592 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.827020 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.827021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:17.827071 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:17.827074 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 01:02:17.827119 (+ 45us) client_negotiation.cc:240] Negotiation successful
0806 01:02:17.827552 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":23,"thread_start_us":93,"threads_started":1}
I20260806 01:02:17.828224 10728 tablet_service.cc:1467] Processing CreateTablet for tablet bb0c8bfac735494187a8591d284df018 (DEFAULT_TABLE table=test-table [id=1e73ab20a19a4ff9bf6d1642121ef0c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:02:17.828287 11008 tablet_service.cc:1467] Processing CreateTablet for tablet bb0c8bfac735494187a8591d284df018 (DEFAULT_TABLE table=test-table [id=1e73ab20a19a4ff9bf6d1642121ef0c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:02:17.828542 10728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb0c8bfac735494187a8591d284df018. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:17.828617 11008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb0c8bfac735494187a8591d284df018. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:17.831295 11099 tablet_bootstrap.cc:492] T bb0c8bfac735494187a8591d284df018 P ec5b0efaf235436b81a28cd4320c3a7a: Bootstrap starting.
I20260806 01:02:17.831955 11099 tablet_bootstrap.cc:654] T bb0c8bfac735494187a8591d284df018 P ec5b0efaf235436b81a28cd4320c3a7a: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:17.831969 11101 tablet_bootstrap.cc:492] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8: Bootstrap starting.
I20260806 01:02:17.832079 11100 tablet_bootstrap.cc:492] T bb0c8bfac735494187a8591d284df018 P dc32bda386c34f228097d138403f8024: Bootstrap starting.
I20260806 01:02:17.832623 11099 log.cc:824] T bb0c8bfac735494187a8591d284df018 P ec5b0efaf235436b81a28cd4320c3a7a: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:17.832630 11101 tablet_bootstrap.cc:654] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:17.832683 11100 tablet_bootstrap.cc:654] T bb0c8bfac735494187a8591d284df018 P dc32bda386c34f228097d138403f8024: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:17.833294 11101 log.cc:824] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:17.833292 11100 log.cc:824] T bb0c8bfac735494187a8591d284df018 P dc32bda386c34f228097d138403f8024: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:17.834223 11101 tablet_bootstrap.cc:492] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8: No bootstrap required, opened a new log
I20260806 01:02:17.834281 11099 tablet_bootstrap.cc:492] T bb0c8bfac735494187a8591d284df018 P ec5b0efaf235436b81a28cd4320c3a7a: No bootstrap required, opened a new log
I20260806 01:02:17.834352 11099 ts_tablet_manager.cc:1397] T bb0c8bfac735494187a8591d284df018 P ec5b0efaf235436b81a28cd4320c3a7a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 01:02:17.834347 11101 ts_tablet_manager.cc:1397] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 01:02:17.834955 11100 tablet_bootstrap.cc:492] T bb0c8bfac735494187a8591d284df018 P dc32bda386c34f228097d138403f8024: No bootstrap required, opened a new log
I20260806 01:02:17.835057 11100 ts_tablet_manager.cc:1397] T bb0c8bfac735494187a8591d284df018 P dc32bda386c34f228097d138403f8024: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 01:02:17.836829 11100 raft_consensus.cc:348] T bb0c8bfac735494187a8591d284df018 P dc32bda386c34f228097d138403f8024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc32bda386c34f228097d138403f8024" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41719 } } peers { permanent_uuid: "b39b7491cd424f3c9dd933dba02b93d8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 41411 } } peers { permanent_uuid: "ec5b0efaf235436b81a28cd4320c3a7a" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 38291 } }
I20260806 01:02:17.837184 11100 raft_consensus.cc:374] T bb0c8bfac735494187a8591d284df018 P dc32bda386c34f228097d138403f8024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:17.837312 11100 raft_consensus.cc:729] T bb0c8bfac735494187a8591d284df018 P dc32bda386c34f228097d138403f8024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc32bda386c34f228097d138403f8024, State: Initialized, Role: FOLLOWER
I20260806 01:02:17.837188 11101 raft_consensus.cc:348] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc32bda386c34f228097d138403f8024" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41719 } } peers { permanent_uuid: "b39b7491cd424f3c9dd933dba02b93d8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 41411 } } peers { permanent_uuid: "ec5b0efaf235436b81a28cd4320c3a7a" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 38291 } }
I20260806 01:02:17.837188 11099 raft_consensus.cc:348] T bb0c8bfac735494187a8591d284df018 P ec5b0efaf235436b81a28cd4320c3a7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc32bda386c34f228097d138403f8024" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41719 } } peers { permanent_uuid: "b39b7491cd424f3c9dd933dba02b93d8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 41411 } } peers { permanent_uuid: "ec5b0efaf235436b81a28cd4320c3a7a" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 38291 } }
I20260806 01:02:17.837414 11099 raft_consensus.cc:374] T bb0c8bfac735494187a8591d284df018 P ec5b0efaf235436b81a28cd4320c3a7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:17.837414 11101 raft_consensus.cc:374] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:17.837464 11101 raft_consensus.cc:729] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b39b7491cd424f3c9dd933dba02b93d8, State: Initialized, Role: FOLLOWER
I20260806 01:02:17.837464 11099 raft_consensus.cc:729] T bb0c8bfac735494187a8591d284df018 P ec5b0efaf235436b81a28cd4320c3a7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec5b0efaf235436b81a28cd4320c3a7a, State: Initialized, Role: FOLLOWER
I20260806 01:02:17.837471 11100 consensus_queue.cc:260] T bb0c8bfac735494187a8591d284df018 P dc32bda386c34f228097d138403f8024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc32bda386c34f228097d138403f8024" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41719 } } peers { permanent_uuid: "b39b7491cd424f3c9dd933dba02b93d8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 41411 } } peers { permanent_uuid: "ec5b0efaf235436b81a28cd4320c3a7a" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 38291 } }
I20260806 01:02:17.837601 11101 consensus_queue.cc:260] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc32bda386c34f228097d138403f8024" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41719 } } peers { permanent_uuid: "b39b7491cd424f3c9dd933dba02b93d8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 41411 } } peers { permanent_uuid: "ec5b0efaf235436b81a28cd4320c3a7a" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 38291 } }
I20260806 01:02:17.837647 11099 consensus_queue.cc:260] T bb0c8bfac735494187a8591d284df018 P ec5b0efaf235436b81a28cd4320c3a7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc32bda386c34f228097d138403f8024" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41719 } } peers { permanent_uuid: "b39b7491cd424f3c9dd933dba02b93d8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 41411 } } peers { permanent_uuid: "ec5b0efaf235436b81a28cd4320c3a7a" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 38291 } }
I20260806 01:02:17.837863 11100 ts_tablet_manager.cc:1428] T bb0c8bfac735494187a8591d284df018 P dc32bda386c34f228097d138403f8024: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 01:02:17.837913 10936 heartbeater.cc:503] Master 127.23.242.62:45707 was elected leader, sending a full tablet report...
I20260806 01:02:17.837942 11101 ts_tablet_manager.cc:1428] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260806 01:02:17.837994 11099 ts_tablet_manager.cc:1428] T bb0c8bfac735494187a8591d284df018 P ec5b0efaf235436b81a28cd4320c3a7a: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260806 01:02:17.838095 11080 heartbeater.cc:503] Master 127.23.242.62:45707 was elected leader, sending a full tablet report...
I20260806 01:02:17.838156 10798 heartbeater.cc:503] Master 127.23.242.62:45707 was elected leader, sending a full tablet report...
W20260806 01:02:17.838938 10937 tablet.cc:2369] T bb0c8bfac735494187a8591d284df018 P dc32bda386c34f228097d138403f8024: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 01:02:17.882346 10799 tablet.cc:2369] T bb0c8bfac735494187a8591d284df018 P ec5b0efaf235436b81a28cd4320c3a7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 01:02:17.887408 11106 raft_consensus.cc:482] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 01:02:17.887614 11106 raft_consensus.cc:504] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc32bda386c34f228097d138403f8024" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41719 } } peers { permanent_uuid: "b39b7491cd424f3c9dd933dba02b93d8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 41411 } } peers { permanent_uuid: "ec5b0efaf235436b81a28cd4320c3a7a" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 38291 } }
I20260806 01:02:17.888075 11106 leader_election.cc:288] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc32bda386c34f228097d138403f8024 (127.23.242.2:41719), ec5b0efaf235436b81a28cd4320c3a7a (127.23.242.1:38291)
I20260806 01:02:17.892196 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:17.888351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:38291 (local address 127.23.242.3:57775)
0806 01:02:17.888401 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:17.888417 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:17.888473 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:17.888795 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:17.888796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:17.888799 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:17.888903 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:17.888906 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.890217 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.890218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:17.891240 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:17.891243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.891893 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.891895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:17.891983 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:17.891987 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 01:02:17.892063 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 01:02:17.892110 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260806 01:02:17.892196 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:17.888427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:57775 (local address 127.23.242.1:38291)
0806 01:02:17.888483 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:17.888485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:17.888500 (+ 15us) server_negotiation.cc:407] Connection header received
0806 01:02:17.888536 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:17.888537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:17.888547 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:17.888785 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:17.889017 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.889791 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.891354 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.891781 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.891881 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:17.891892 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:17.892070 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 01:02:17.892112 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 01:02:17.892350 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:17.888507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:37827 (local address 127.23.242.2:41719)
0806 01:02:17.888574 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:17.888578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:17.889131 (+ 553us) server_negotiation.cc:407] Connection header received
0806 01:02:17.889169 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:17.889170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:17.889181 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:17.889260 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:17.889455 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.890386 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.891502 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.891936 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.892037 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:17.892046 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:17.892219 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 01:02:17.892263 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 01:02:17.892611 10748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb0c8bfac735494187a8591d284df018" candidate_uuid: "b39b7491cd424f3c9dd933dba02b93d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec5b0efaf235436b81a28cd4320c3a7a" is_pre_election: true
I20260806 01:02:17.892765 10748 raft_consensus.cc:2454] T bb0c8bfac735494187a8591d284df018 P ec5b0efaf235436b81a28cd4320c3a7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b39b7491cd424f3c9dd933dba02b93d8 in term 0.
I20260806 01:02:17.892938 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:17.888353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:41719 (local address 127.23.242.3:37827)
0806 01:02:17.889032 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:17.889042 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:17.889100 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:17.889299 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:17.889300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:17.889302 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:17.889365 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:17.889366 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.890485 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.890486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:17.891393 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:17.891395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.892037 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.892039 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:17.892125 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:17.892134 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 01:02:17.892212 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 01:02:17.892255 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":476,"thread_start_us":130,"threads_started":1}
I20260806 01:02:17.893141 10960 leader_election.cc:302] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b39b7491cd424f3c9dd933dba02b93d8, ec5b0efaf235436b81a28cd4320c3a7a; no voters:
I20260806 01:02:17.893306 11106 raft_consensus.cc:2781] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 01:02:17.893440 10889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb0c8bfac735494187a8591d284df018" candidate_uuid: "b39b7491cd424f3c9dd933dba02b93d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc32bda386c34f228097d138403f8024" is_pre_election: true
I20260806 01:02:17.893404 11106 raft_consensus.cc:482] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 01:02:17.893565 11106 raft_consensus.cc:3037] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:17.893620 10889 raft_consensus.cc:2454] T bb0c8bfac735494187a8591d284df018 P dc32bda386c34f228097d138403f8024 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b39b7491cd424f3c9dd933dba02b93d8 in term 0.
I20260806 01:02:17.894479 11106 raft_consensus.cc:504] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc32bda386c34f228097d138403f8024" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41719 } } peers { permanent_uuid: "b39b7491cd424f3c9dd933dba02b93d8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 41411 } } peers { permanent_uuid: "ec5b0efaf235436b81a28cd4320c3a7a" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 38291 } }
I20260806 01:02:17.894687 11106 leader_election.cc:288] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8 [CANDIDATE]: Term 1 election: Requested vote from peers dc32bda386c34f228097d138403f8024 (127.23.242.2:41719), ec5b0efaf235436b81a28cd4320c3a7a (127.23.242.1:38291)
I20260806 01:02:17.894985 10748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb0c8bfac735494187a8591d284df018" candidate_uuid: "b39b7491cd424f3c9dd933dba02b93d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec5b0efaf235436b81a28cd4320c3a7a"
I20260806 01:02:17.894976 10889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb0c8bfac735494187a8591d284df018" candidate_uuid: "b39b7491cd424f3c9dd933dba02b93d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc32bda386c34f228097d138403f8024"
I20260806 01:02:17.895071 10889 raft_consensus.cc:3037] T bb0c8bfac735494187a8591d284df018 P dc32bda386c34f228097d138403f8024 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:17.895074 10748 raft_consensus.cc:3037] T bb0c8bfac735494187a8591d284df018 P ec5b0efaf235436b81a28cd4320c3a7a [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:17.896008 10889 raft_consensus.cc:2454] T bb0c8bfac735494187a8591d284df018 P dc32bda386c34f228097d138403f8024 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b39b7491cd424f3c9dd933dba02b93d8 in term 1.
I20260806 01:02:17.896008 10748 raft_consensus.cc:2454] T bb0c8bfac735494187a8591d284df018 P ec5b0efaf235436b81a28cd4320c3a7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b39b7491cd424f3c9dd933dba02b93d8 in term 1.
I20260806 01:02:17.896214 10960 leader_election.cc:302] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b39b7491cd424f3c9dd933dba02b93d8, ec5b0efaf235436b81a28cd4320c3a7a; no voters:
I20260806 01:02:17.896322 11106 raft_consensus.cc:2781] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:17.896499 11106 raft_consensus.cc:686] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8 [term 1 LEADER]: Becoming Leader. State: Replica: b39b7491cd424f3c9dd933dba02b93d8, State: Running, Role: LEADER
I20260806 01:02:17.896616 11106 consensus_queue.cc:237] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc32bda386c34f228097d138403f8024" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41719 } } peers { permanent_uuid: "b39b7491cd424f3c9dd933dba02b93d8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 41411 } } peers { permanent_uuid: "ec5b0efaf235436b81a28cd4320c3a7a" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 38291 } }
I20260806 01:02:17.897940 10598 catalog_manager.cc:5495] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b39b7491cd424f3c9dd933dba02b93d8 (127.23.242.3). New cstate: current_term: 1 leader_uuid: "b39b7491cd424f3c9dd933dba02b93d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc32bda386c34f228097d138403f8024" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b39b7491cd424f3c9dd933dba02b93d8" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 41411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec5b0efaf235436b81a28cd4320c3a7a" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 38291 } health_report { overall_health: UNKNOWN } } }
I20260806 01:02:17.923748 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:17.920520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56652 (local address 127.23.242.3:41411)
0806 01:02:17.920665 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:17.920669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:17.920793 (+ 124us) server_negotiation.cc:407] Connection header received
0806 01:02:17.921003 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:17.921005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:17.921045 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:17.921195 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 01:02:17.921564 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.922095 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.922914 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:17.923108 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:17.923196 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:17.923364 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 01:02:17.923464 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 01:02:17.923578 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:17.923610 (+ 32us) server_negotiation.cc:299] Negotiation successful
0806 01:02:17.923622 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260806 01:02:17.926358 10889 raft_consensus.cc:1261] T bb0c8bfac735494187a8591d284df018 P dc32bda386c34f228097d138403f8024 [term 1 FOLLOWER]: Refusing update from remote peer b39b7491cd424f3c9dd933dba02b93d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:02:17.926739 10748 raft_consensus.cc:1261] T bb0c8bfac735494187a8591d284df018 P ec5b0efaf235436b81a28cd4320c3a7a [term 1 FOLLOWER]: Refusing update from remote peer b39b7491cd424f3c9dd933dba02b93d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:02:17.927110 11106 consensus_queue.cc:1035] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc32bda386c34f228097d138403f8024" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 41719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:02:17.927381 11109 consensus_queue.cc:1035] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec5b0efaf235436b81a28cd4320c3a7a" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 38291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:02:17.930290 11113 mvcc.cc:205] Tried to move back new op lower bound from 7315366452942835712 to 7315366452825227264. Current Snapshot: MvccSnapshot[applied={T|T < 7315366452942835712}]
I20260806 01:02:17.930764 11114 mvcc.cc:205] Tried to move back new op lower bound from 7315366452942835712 to 7315366452825227264. Current Snapshot: MvccSnapshot[applied={T|T < 7315366452942835712}]
I20260806 01:02:17.930908 11112 mvcc.cc:205] Tried to move back new op lower bound from 7315366452942835712 to 7315366452825227264. Current Snapshot: MvccSnapshot[applied={T|T < 7315366452942835712}]
I20260806 01:02:17.933621 10596 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 01:02:17.933751 10596 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 01:02:17.934777 10596 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a00713e064f142b883632e34251a1271: Sending DeleteTablet for 3 replicas of tablet bb0c8bfac735494187a8591d284df018
I20260806 01:02:17.935222 11008 tablet_service.cc:1514] Processing DeleteTablet for tablet bb0c8bfac735494187a8591d284df018 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:02:17 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:56642
I20260806 01:02:17.935259 10869 tablet_service.cc:1514] Processing DeleteTablet for tablet bb0c8bfac735494187a8591d284df018 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:02:17 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:57642
I20260806 01:02:17.935278 10728 tablet_service.cc:1514] Processing DeleteTablet for tablet bb0c8bfac735494187a8591d284df018 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:02:17 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:41526
I20260806 01:02:17.935523 11124 tablet_replica.cc:330] T bb0c8bfac735494187a8591d284df018 P ec5b0efaf235436b81a28cd4320c3a7a: stopping tablet replica
I20260806 01:02:17.935640 11124 raft_consensus.cc:2229] T bb0c8bfac735494187a8591d284df018 P ec5b0efaf235436b81a28cd4320c3a7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 01:02:17.935770 11124 raft_consensus.cc:2258] T bb0c8bfac735494187a8591d284df018 P ec5b0efaf235436b81a28cd4320c3a7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 01:02:17.935942 11122 tablet_replica.cc:330] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8: stopping tablet replica
I20260806 01:02:17.936036 11122 raft_consensus.cc:2229] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8 [term 1 LEADER]: Raft consensus shutting down.
I20260806 01:02:17.936160 11122 raft_consensus.cc:2258] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 01:02:17.936245 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 10661
I20260806 01:02:17.936322 11123 tablet_replica.cc:330] T bb0c8bfac735494187a8591d284df018 P dc32bda386c34f228097d138403f8024: stopping tablet replica
I20260806 01:02:17.936414 11123 raft_consensus.cc:2229] T bb0c8bfac735494187a8591d284df018 P dc32bda386c34f228097d138403f8024 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 01:02:17.936496 11123 raft_consensus.cc:2258] T bb0c8bfac735494187a8591d284df018 P dc32bda386c34f228097d138403f8024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 01:02:17.937597 11123 ts_tablet_manager.cc:1905] T bb0c8bfac735494187a8591d284df018 P dc32bda386c34f228097d138403f8024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 01:02:17.939769 11122 ts_tablet_manager.cc:1905] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 01:02:17.940301 11123 ts_tablet_manager.cc:1918] T bb0c8bfac735494187a8591d284df018 P dc32bda386c34f228097d138403f8024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 01:02:17.940397 11123 log.cc:1196] T bb0c8bfac735494187a8591d284df018 P dc32bda386c34f228097d138403f8024: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-1/wal/wals/bb0c8bfac735494187a8591d284df018
I20260806 01:02:17.940761 11123 ts_tablet_manager.cc:1939] T bb0c8bfac735494187a8591d284df018 P dc32bda386c34f228097d138403f8024: Deleting consensus metadata
I20260806 01:02:17.941283 10582 catalog_manager.cc:4841] TS dc32bda386c34f228097d138403f8024 (127.23.242.2:41719): tablet bb0c8bfac735494187a8591d284df018 (table test-table [id=1e73ab20a19a4ff9bf6d1642121ef0c2]) successfully deleted
I20260806 01:02:17.942291 11122 ts_tablet_manager.cc:1918] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 01:02:17.942441 11122 log.cc:1196] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785977869987768-24520-0/minicluster-data/ts-2/wal/wals/bb0c8bfac735494187a8591d284df018
I20260806 01:02:17.942839 11122 ts_tablet_manager.cc:1939] T bb0c8bfac735494187a8591d284df018 P b39b7491cd424f3c9dd933dba02b93d8: Deleting consensus metadata
I20260806 01:02:17.943450 10584 catalog_manager.cc:4841] TS b39b7491cd424f3c9dd933dba02b93d8 (127.23.242.3:41411): tablet bb0c8bfac735494187a8591d284df018 (table test-table [id=1e73ab20a19a4ff9bf6d1642121ef0c2]) successfully deleted
W20260806 01:02:17.944579 10583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.242.1:38291 (error 108)
I20260806 01:02:17.944895 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 10803
I20260806 01:02:17.948994 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:17.948811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:38291 (local address 127.0.0.1:41540)
0806 01:02:17.948879 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:17.948940 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.1:38291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 01:02:17.949478 10583 catalog_manager.cc:4584] TS ec5b0efaf235436b81a28cd4320c3a7a (127.23.242.1:38291): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bb0c8bfac735494187a8591d284df018: Network error: Client connection negotiation failed: client connection to 127.23.242.1:38291: connect: Connection refused (error 111)
I20260806 01:02:17.954036 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 10943
I20260806 01:02:17.961412 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 10566
2026-08-06T01:02:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2509 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-06T01:02:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:02:17Z Disabled control of system clock
I20260806 01:02:17.994832 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:37409
--webserver_interface=127.23.242.62
--webserver_port=0
--builtin_ntp_servers=127.23.242.20:38647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:37409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 01:02:18.096920 11130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:18.097190 11130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:18.097215 11130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:18.098860 11130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:18.098932 11130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:18.098948 11130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:18.098963 11130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:18.098978 11130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:18.100621 11130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:38647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:37409
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:37409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:18.100937 11130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:18.101323 11130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:18.107342 11135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:18.107344 11138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:18.107344 11136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:18.107566 11130 server_base.cc:1034] running on GCE node
I20260806 01:02:18.107956 11130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:18.108247 11130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:18.109443 11130 hybrid_clock.cc:648] HybridClock initialized: now 1785978138109407 us; error 55 us; skew 500 ppm
I20260806 01:02:18.111189 11130 webserver.cc:460] Webserver started at http://127.23.242.62:42011/ using document root <none> and password file <none>
I20260806 01:02:18.111526 11130 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:18.111608 11130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:18.111789 11130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:18.113102 11130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "e56fb3454f9c405d8c27bd6bd2b9638d"
format_stamp: "Formatted at 2026-08-06 01:02:18 on dist-test-slave-dhph"
server_key: "1ff31932e0e77ddb413fb1d538d31c56"
server_key_iv: "643d7764a027ace576445883931bb7d7"
server_key_version: "encryptionkey@0"
I20260806 01:02:18.113519 11130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "e56fb3454f9c405d8c27bd6bd2b9638d"
format_stamp: "Formatted at 2026-08-06 01:02:18 on dist-test-slave-dhph"
server_key: "1ff31932e0e77ddb413fb1d538d31c56"
server_key_iv: "643d7764a027ace576445883931bb7d7"
server_key_version: "encryptionkey@0"
I20260806 01:02:18.116206 11130 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.005s
I20260806 01:02:18.117415 11147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:18.117709 11130 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 01:02:18.117828 11130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "e56fb3454f9c405d8c27bd6bd2b9638d"
format_stamp: "Formatted at 2026-08-06 01:02:18 on dist-test-slave-dhph"
server_key: "1ff31932e0e77ddb413fb1d538d31c56"
server_key_iv: "643d7764a027ace576445883931bb7d7"
server_key_version: "encryptionkey@0"
I20260806 01:02:18.117949 11130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:18.136129 11130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:18.136543 11130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:18.136806 11130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:18.142112 11130 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:37409
I20260806 01:02:18.142192 11201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:37409 every 8 connection(s)
I20260806 01:02:18.142706 11130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:02:18.143852 11202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:18.146756 11202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e56fb3454f9c405d8c27bd6bd2b9638d: Bootstrap starting.
I20260806 01:02:18.147397 11202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e56fb3454f9c405d8c27bd6bd2b9638d: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:18.147753 11202 log.cc:824] T 00000000000000000000000000000000 P e56fb3454f9c405d8c27bd6bd2b9638d: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:18.148602 11202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e56fb3454f9c405d8c27bd6bd2b9638d: No bootstrap required, opened a new log
I20260806 01:02:18.150553 11202 raft_consensus.cc:348] T 00000000000000000000000000000000 P e56fb3454f9c405d8c27bd6bd2b9638d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56fb3454f9c405d8c27bd6bd2b9638d" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37409 } }
I20260806 01:02:18.150748 11202 raft_consensus.cc:374] T 00000000000000000000000000000000 P e56fb3454f9c405d8c27bd6bd2b9638d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:18.150777 11202 raft_consensus.cc:729] T 00000000000000000000000000000000 P e56fb3454f9c405d8c27bd6bd2b9638d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e56fb3454f9c405d8c27bd6bd2b9638d, State: Initialized, Role: FOLLOWER
I20260806 01:02:18.150955 11202 consensus_queue.cc:260] T 00000000000000000000000000000000 P e56fb3454f9c405d8c27bd6bd2b9638d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56fb3454f9c405d8c27bd6bd2b9638d" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37409 } }
I20260806 01:02:18.151044 11202 raft_consensus.cc:388] T 00000000000000000000000000000000 P e56fb3454f9c405d8c27bd6bd2b9638d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:02:18.151070 11202 raft_consensus.cc:482] T 00000000000000000000000000000000 P e56fb3454f9c405d8c27bd6bd2b9638d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:02:18.151126 11202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e56fb3454f9c405d8c27bd6bd2b9638d [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:18.151996 11202 raft_consensus.cc:504] T 00000000000000000000000000000000 P e56fb3454f9c405d8c27bd6bd2b9638d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56fb3454f9c405d8c27bd6bd2b9638d" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37409 } }
I20260806 01:02:18.152100 11202 leader_election.cc:302] T 00000000000000000000000000000000 P e56fb3454f9c405d8c27bd6bd2b9638d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e56fb3454f9c405d8c27bd6bd2b9638d; no voters:
I20260806 01:02:18.152309 11202 leader_election.cc:288] T 00000000000000000000000000000000 P e56fb3454f9c405d8c27bd6bd2b9638d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:02:18.152427 11205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e56fb3454f9c405d8c27bd6bd2b9638d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:18.152632 11205 raft_consensus.cc:686] T 00000000000000000000000000000000 P e56fb3454f9c405d8c27bd6bd2b9638d [term 1 LEADER]: Becoming Leader. State: Replica: e56fb3454f9c405d8c27bd6bd2b9638d, State: Running, Role: LEADER
I20260806 01:02:18.152640 11202 sys_catalog.cc:564] T 00000000000000000000000000000000 P e56fb3454f9c405d8c27bd6bd2b9638d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:18.152810 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 11130
I20260806 01:02:18.152750 11205 consensus_queue.cc:237] T 00000000000000000000000000000000 P e56fb3454f9c405d8c27bd6bd2b9638d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56fb3454f9c405d8c27bd6bd2b9638d" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37409 } }
I20260806 01:02:18.152902 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:02:18.153134 24520 external_mini_cluster.cc:1428] Setting key 35d93318cacd57f16b159bff12f9367c
I20260806 01:02:18.153299 11207 sys_catalog.cc:455] T 00000000000000000000000000000000 P e56fb3454f9c405d8c27bd6bd2b9638d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e56fb3454f9c405d8c27bd6bd2b9638d. Latest consensus state: current_term: 1 leader_uuid: "e56fb3454f9c405d8c27bd6bd2b9638d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56fb3454f9c405d8c27bd6bd2b9638d" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37409 } } }
I20260806 01:02:18.153412 11207 sys_catalog.cc:458] T 00000000000000000000000000000000 P e56fb3454f9c405d8c27bd6bd2b9638d [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:18.153394 11206 sys_catalog.cc:455] T 00000000000000000000000000000000 P e56fb3454f9c405d8c27bd6bd2b9638d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e56fb3454f9c405d8c27bd6bd2b9638d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56fb3454f9c405d8c27bd6bd2b9638d" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37409 } } }
I20260806 01:02:18.153474 11206 sys_catalog.cc:458] T 00000000000000000000000000000000 P e56fb3454f9c405d8c27bd6bd2b9638d [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:18.153651 11213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:02:18.154242 11213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:02:18.156337 11213 catalog_manager.cc:1347] Generated new cluster ID: dff426a607c8449f8c3691259e227653
I20260806 01:02:18.156406 11213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:02:18.161118 11216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:18.153829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41882 (local address 127.23.242.62:37409)
0806 01:02:18.154467 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:18.154476 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:18.154919 (+ 443us) server_negotiation.cc:407] Connection header received
0806 01:02:18.155062 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:18.155072 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:18.155156 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:18.155342 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:18.155629 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.157322 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.158640 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.159337 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.159466 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:18.160329 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:18.160333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:18.160334 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:18.160554 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:18.160599 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:18.160614 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:18.160687 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:18.160802 (+ 115us) server_negotiation.cc:299] Negotiation successful
0806 01:02:18.160885 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":85,"threads_started":1}
I20260806 01:02:18.179584 11213 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:02:18.180327 11213 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:02:18.184659 11213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e56fb3454f9c405d8c27bd6bd2b9638d: Generated new TSK 0
I20260806 01:02:18.184886 11213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 01:02:18.189823 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:0
--local_ip_for_outbound_sockets=127.23.242.1
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:37409
--builtin_ntp_servers=127.23.242.20:38647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 01:02:18.297479 11227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:18.297739 11227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:18.297768 11227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:18.299841 11227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:18.299922 11227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:18.299958 11227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 01:02:18.301708 11227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:38647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:37409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:18.302038 11227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:18.302423 11227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:18.309615 11232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:18.309629 11233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:18.309715 11235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:18.310827 11227 server_base.cc:1034] running on GCE node
I20260806 01:02:18.311093 11227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:18.311399 11227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:18.312644 11227 hybrid_clock.cc:648] HybridClock initialized: now 1785978138312612 us; error 52 us; skew 500 ppm
I20260806 01:02:18.314682 11227 webserver.cc:460] Webserver started at http://127.23.242.1:42075/ using document root <none> and password file <none>
I20260806 01:02:18.315033 11227 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:18.315127 11227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:18.315310 11227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:18.316574 11227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-0/data/instance:
uuid: "23c299bf9cd34d5fbe8376cb40dd9fe5"
format_stamp: "Formatted at 2026-08-06 01:02:18 on dist-test-slave-dhph"
server_key: "c0ed279eaad1544035686e41955cec35"
server_key_iv: "c2068087bec09d9730a22e603c9266b1"
server_key_version: "encryptionkey@0"
I20260806 01:02:18.317021 11227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance:
uuid: "23c299bf9cd34d5fbe8376cb40dd9fe5"
format_stamp: "Formatted at 2026-08-06 01:02:18 on dist-test-slave-dhph"
server_key: "c0ed279eaad1544035686e41955cec35"
server_key_iv: "c2068087bec09d9730a22e603c9266b1"
server_key_version: "encryptionkey@0"
I20260806 01:02:18.318822 11227 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260806 01:02:18.319854 11241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:18.320247 11227 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:18.320394 11227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "23c299bf9cd34d5fbe8376cb40dd9fe5"
format_stamp: "Formatted at 2026-08-06 01:02:18 on dist-test-slave-dhph"
server_key: "c0ed279eaad1544035686e41955cec35"
server_key_iv: "c2068087bec09d9730a22e603c9266b1"
server_key_version: "encryptionkey@0"
I20260806 01:02:18.320498 11227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:18.334128 11227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:18.334524 11227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:18.334779 11227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:18.335075 11227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:02:18.335453 11227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:02:18.335538 11227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:18.335578 11227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:02:18.335633 11227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:18.345161 11227 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:44045
I20260806 01:02:18.345278 11358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:44045 every 8 connection(s)
I20260806 01:02:18.345687 11227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 01:02:18.348098 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 11227
I20260806 01:02:18.348261 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance
I20260806 01:02:18.348613 24520 external_mini_cluster.cc:1428] Setting key eac70db480fb7e6a1f42446bbf76c61f
I20260806 01:02:18.350766 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:0
--local_ip_for_outbound_sockets=127.23.242.2
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:37409
--builtin_ntp_servers=127.23.242.20:38647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 01:02:18.354645 11216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:18.346634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:48873 (local address 127.23.242.62:37409)
0806 01:02:18.346698 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:18.346700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:18.346983 (+ 283us) server_negotiation.cc:407] Connection header received
0806 01:02:18.347482 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:18.347483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:18.347504 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:18.347595 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:18.349159 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.349717 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.352073 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.352476 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.352573 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:18.354123 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:18.354124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:18.354125 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:18.354132 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:18.354150 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:18.354156 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:18.354218 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:18.354451 (+ 233us) server_negotiation.cc:299] Negotiation successful
0806 01:02:18.354489 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 01:02:18.354835 11361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:18.346544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:37409 (local address 127.23.242.1:48873)
0806 01:02:18.346767 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:18.346812 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:18.347352 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:18.347699 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:18.347703 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:18.347711 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:18.348742 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:18.348750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.349882 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.349885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:18.351782 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:18.351788 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.352670 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.352671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:18.352940 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:18.353931 (+ 991us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 01:02:18.353932 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:18.353976 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 01:02:18.354011 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:18.354347 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:18.354354 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:18.354357 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 01:02:18.354435 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 01:02:18.354535 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":116,"threads_started":1}
I20260806 01:02:18.355438 11359 heartbeater.cc:347] Connected to a master server at 127.23.242.62:37409
I20260806 01:02:18.355572 11359 heartbeater.cc:464] Registering TS with master...
I20260806 01:02:18.355796 11359 heartbeater.cc:511] Master 127.23.242.62:37409 requested a full tablet report, sending...
I20260806 01:02:18.356438 11164 ts_manager.cc:194] Registered new tserver with Master: 23c299bf9cd34d5fbe8376cb40dd9fe5 (127.23.242.1:44045)
I20260806 01:02:18.357075 11164 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.242.1:48873
W20260806 01:02:18.449918 11362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:18.450191 11362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:18.450230 11362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:18.451560 11362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:18.451642 11362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:18.451690 11362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 01:02:18.453485 11362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:38647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:37409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:18.453792 11362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:18.454191 11362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:18.461046 11368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:18.461067 11371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:18.461092 11369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:18.461429 11362 server_base.cc:1034] running on GCE node
I20260806 01:02:18.461684 11362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:18.462019 11362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:18.463194 11362 hybrid_clock.cc:648] HybridClock initialized: now 1785978138463150 us; error 62 us; skew 500 ppm
I20260806 01:02:18.465263 11362 webserver.cc:460] Webserver started at http://127.23.242.2:44217/ using document root <none> and password file <none>
I20260806 01:02:18.465700 11362 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:18.465777 11362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:18.465974 11362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:18.467366 11362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-1/data/instance:
uuid: "20c06efe6f604539900f0458b56bb974"
format_stamp: "Formatted at 2026-08-06 01:02:18 on dist-test-slave-dhph"
server_key: "0363d223915dd9e7ce9969fa5418afc8"
server_key_iv: "0e4a4f452876304c97fe76a9ed265b25"
server_key_version: "encryptionkey@0"
I20260806 01:02:18.467773 11362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance:
uuid: "20c06efe6f604539900f0458b56bb974"
format_stamp: "Formatted at 2026-08-06 01:02:18 on dist-test-slave-dhph"
server_key: "0363d223915dd9e7ce9969fa5418afc8"
server_key_iv: "0e4a4f452876304c97fe76a9ed265b25"
server_key_version: "encryptionkey@0"
I20260806 01:02:18.469530 11362 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 01:02:18.470643 11377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:18.470886 11362 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:18.471019 11362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "20c06efe6f604539900f0458b56bb974"
format_stamp: "Formatted at 2026-08-06 01:02:18 on dist-test-slave-dhph"
server_key: "0363d223915dd9e7ce9969fa5418afc8"
server_key_iv: "0e4a4f452876304c97fe76a9ed265b25"
server_key_version: "encryptionkey@0"
I20260806 01:02:18.471113 11362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:18.491425 11362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:18.491839 11362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:18.492025 11362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:18.492309 11362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:02:18.493038 11362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:02:18.493170 11362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:18.493271 11362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:02:18.493305 11362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:18.502944 11362 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:38257
I20260806 01:02:18.503085 11495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:38257 every 8 connection(s)
I20260806 01:02:18.503456 11362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 01:02:18.509719 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 11362
I20260806 01:02:18.509887 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance
I20260806 01:02:18.510200 24520 external_mini_cluster.cc:1428] Setting key 2949f809bb77f3cde4b343d07e3285e2
I20260806 01:02:18.510748 11216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:18.504431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:37045 (local address 127.23.242.62:37409)
0806 01:02:18.504521 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:18.504524 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:18.504833 (+ 309us) server_negotiation.cc:407] Connection header received
0806 01:02:18.505412 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:18.505413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:18.505424 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:18.505486 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:18.506452 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.507081 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.508616 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.508835 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.508925 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:18.510110 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:18.510112 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:18.510113 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:18.510163 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:18.510183 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:18.510188 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:18.510276 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:18.510616 (+ 340us) server_negotiation.cc:299] Negotiation successful
0806 01:02:18.510655 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 01:02:18.511224 11498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:18.504411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:37409 (local address 127.23.242.2:37045)
0806 01:02:18.504669 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:18.504707 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:18.505284 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:18.505706 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:18.505709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:18.505719 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:18.506304 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:18.506309 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.507286 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.507288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:18.508468 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:18.508472 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.508973 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.508980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:18.509115 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:18.509854 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 01:02:18.509855 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:18.509924 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 01:02:18.509960 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:18.510374 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:18.510392 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:18.510398 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 01:02:18.510621 (+ 223us) client_negotiation.cc:240] Negotiation successful
0806 01:02:18.510884 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":117,"threads_started":1}
I20260806 01:02:18.511757 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:0
--local_ip_for_outbound_sockets=127.23.242.3
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:37409
--builtin_ntp_servers=127.23.242.20:38647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 01:02:18.512169 11496 heartbeater.cc:347] Connected to a master server at 127.23.242.62:37409
I20260806 01:02:18.512372 11496 heartbeater.cc:464] Registering TS with master...
I20260806 01:02:18.512708 11496 heartbeater.cc:511] Master 127.23.242.62:37409 requested a full tablet report, sending...
I20260806 01:02:18.513278 11164 ts_manager.cc:194] Registered new tserver with Master: 20c06efe6f604539900f0458b56bb974 (127.23.242.2:38257)
I20260806 01:02:18.513842 11164 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.242.2:37045
W20260806 01:02:18.612979 11499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:18.613255 11499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:18.613300 11499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:18.614934 11499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:18.615025 11499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:18.615083 11499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 01:02:18.617015 11499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:38647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:37409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:18.617336 11499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:18.617690 11499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:18.624325 11507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:18.624402 11505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:18.624352 11504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:18.625788 11499 server_base.cc:1034] running on GCE node
I20260806 01:02:18.626219 11499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:18.626540 11499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:18.627965 11499 hybrid_clock.cc:648] HybridClock initialized: now 1785978138627888 us; error 54 us; skew 500 ppm
I20260806 01:02:18.629881 11499 webserver.cc:460] Webserver started at http://127.23.242.3:35267/ using document root <none> and password file <none>
I20260806 01:02:18.630193 11499 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:18.630276 11499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:18.630446 11499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:18.631819 11499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-2/data/instance:
uuid: "793e3e9f04084db8ac69dbe3473515d2"
format_stamp: "Formatted at 2026-08-06 01:02:18 on dist-test-slave-dhph"
server_key: "9b65389d8c90576e6bd2352212e3c603"
server_key_iv: "17e1114d55c172e245df9cc69412a730"
server_key_version: "encryptionkey@0"
I20260806 01:02:18.632249 11499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance:
uuid: "793e3e9f04084db8ac69dbe3473515d2"
format_stamp: "Formatted at 2026-08-06 01:02:18 on dist-test-slave-dhph"
server_key: "9b65389d8c90576e6bd2352212e3c603"
server_key_iv: "17e1114d55c172e245df9cc69412a730"
server_key_version: "encryptionkey@0"
I20260806 01:02:18.633904 11499 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 01:02:18.635030 11513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:18.635277 11499 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 01:02:18.635411 11499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "793e3e9f04084db8ac69dbe3473515d2"
format_stamp: "Formatted at 2026-08-06 01:02:18 on dist-test-slave-dhph"
server_key: "9b65389d8c90576e6bd2352212e3c603"
server_key_iv: "17e1114d55c172e245df9cc69412a730"
server_key_version: "encryptionkey@0"
I20260806 01:02:18.635506 11499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:18.661695 11499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:18.662108 11499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:18.662294 11499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:18.662663 11499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:02:18.663108 11499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:02:18.663173 11499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:18.663225 11499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:02:18.663259 11499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:18.670429 11499 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:38203
I20260806 01:02:18.670477 11637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:38203 every 8 connection(s)
I20260806 01:02:18.671001 11499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
I20260806 01:02:18.678769 11216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:18.672132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:54865 (local address 127.23.242.62:37409)
0806 01:02:18.672221 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:18.672223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:18.672435 (+ 212us) server_negotiation.cc:407] Connection header received
0806 01:02:18.673288 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:18.673289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:18.673307 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:18.673383 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:18.674879 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.675425 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.676623 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.676868 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.676944 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:18.678293 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:18.678295 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:18.678295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:18.678300 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:18.678316 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:18.678320 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:18.678386 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:18.678588 (+ 202us) server_negotiation.cc:299] Negotiation successful
0806 01:02:18.678687 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 01:02:18.679039 11640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:18.672023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:37409 (local address 127.23.242.3:54865)
0806 01:02:18.672248 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:18.672294 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:18.672990 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:18.673625 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:18.673632 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:18.673701 (+ 69us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:18.674686 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:18.674698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.675551 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.675552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:18.676507 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:18.676509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.677059 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.677062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:18.677340 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:18.678103 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 01:02:18.678104 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:18.678146 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 01:02:18.678176 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:18.678427 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:18.678434 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:18.678437 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 01:02:18.678517 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 01:02:18.678741 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":104,"threads_started":1}
I20260806 01:02:18.679414 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 11499
I20260806 01:02:18.679575 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance
I20260806 01:02:18.679728 11638 heartbeater.cc:347] Connected to a master server at 127.23.242.62:37409
I20260806 01:02:18.679847 11638 heartbeater.cc:464] Registering TS with master...
I20260806 01:02:18.679858 24520 external_mini_cluster.cc:1428] Setting key b14f12b7a6ba7d4441f81f0838c9ec29
I20260806 01:02:18.680092 11638 heartbeater.cc:511] Master 127.23.242.62:37409 requested a full tablet report, sending...
I20260806 01:02:18.680622 11164 ts_manager.cc:194] Registered new tserver with Master: 793e3e9f04084db8ac69dbe3473515d2 (127.23.242.3:38203)
I20260806 01:02:18.681143 11164 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.242.3:54865
I20260806 01:02:18.681169 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 01:02:18.687211 11216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:18.682931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41892 (local address 127.23.242.62:37409)
0806 01:02:18.683045 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:18.683048 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:18.683150 (+ 102us) server_negotiation.cc:407] Connection header received
0806 01:02:18.683228 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:18.683229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:18.683240 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:18.683297 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:18.683483 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.684511 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.685641 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.685876 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.685951 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:18.686884 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:18.686885 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:18.686885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:18.686890 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:18.686903 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:18.686907 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:18.686987 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:18.687133 (+ 146us) server_negotiation.cc:299] Negotiation successful
0806 01:02:18.687144 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 01:02:18.689407 11164 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 01:02:18.691296 11164 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 01:02:18.699108 11654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:18.694419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33314 (local address 127.23.242.2:38257)
0806 01:02:18.695208 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:18.695212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:18.695223 (+ 11us) server_negotiation.cc:407] Connection header received
0806 01:02:18.695264 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:18.695266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:18.695281 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:18.695394 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:18.695518 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.696377 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.697754 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.698589 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.698687 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:18.698788 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:18.698950 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 01:02:18.698997 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":85,"threads_started":1}
I20260806 01:02:18.699088 11651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:18.694128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:38257 (local address 127.0.0.1:33314)
0806 01:02:18.694653 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:18.694665 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:18.694727 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:18.695392 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:18.695393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:18.695396 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:18.695447 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:18.695449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.696516 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.696517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:18.697647 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:18.697650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.698715 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.698717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:18.698817 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:18.698835 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 01:02:18.698921 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 01:02:18.698984 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":6,"thread_start_us":83,"threads_started":1}
I20260806 01:02:18.699724 11649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:18.694132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:44045 (local address 127.0.0.1:55454)
0806 01:02:18.694414 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:18.694456 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:18.694533 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:18.694899 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:18.694902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:18.694905 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:18.695044 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:18.695046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.696765 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.696766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:18.698159 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:18.698171 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.699367 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.699368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:18.699455 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:18.699460 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 01:02:18.699520 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 01:02:18.699663 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":87,"thread_start_us":123,"threads_started":1}
I20260806 01:02:18.700055 11653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:18.694239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55454 (local address 127.23.242.1:44045)
0806 01:02:18.694730 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:18.694735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:18.694744 (+ 9us) server_negotiation.cc:407] Connection header received
0806 01:02:18.694779 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:18.694781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:18.694794 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:18.694893 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:18.695181 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.696537 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.698264 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.699110 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.699190 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:18.699288 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:18.699916 (+ 628us) server_negotiation.cc:299] Negotiation successful
0806 01:02:18.699968 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":93,"threads_started":1}
I20260806 01:02:18.700070 11428 tablet_service.cc:1467] Processing CreateTablet for tablet d1f422bb8f43498fbd80e808668a4eea (DEFAULT_TABLE table=test-table [id=ca5b5be25bdc4f7cadf23cd5bd82bca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:02:18.700543 11428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1f422bb8f43498fbd80e808668a4eea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:18.700697 11650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:18.694128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:38203 (local address 127.0.0.1:51990)
0806 01:02:18.694408 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:18.694456 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:18.694533 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:18.695071 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:18.695072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:18.695074 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:18.695118 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:18.695119 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.696627 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.696627 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:18.698159 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:18.698163 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.700416 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.700417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:18.700492 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:18.700496 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 01:02:18.700544 (+ 48us) client_negotiation.cc:240] Negotiation successful
0806 01:02:18.700599 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":138,"thread_start_us":83,"threads_started":1}
I20260806 01:02:18.701059 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:18.694419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51990 (local address 127.23.242.3:38203)
0806 01:02:18.694888 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:18.694893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:18.694904 (+ 11us) server_negotiation.cc:407] Connection header received
0806 01:02:18.694949 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:18.694951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:18.694969 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:18.695072 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:18.695216 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.696528 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.698722 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.700231 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.700316 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:18.700876 (+ 560us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:18.700908 (+ 32us) server_negotiation.cc:299] Negotiation successful
0806 01:02:18.700962 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":92,"threads_started":1}
I20260806 01:02:18.702126 11563 tablet_service.cc:1467] Processing CreateTablet for tablet d1f422bb8f43498fbd80e808668a4eea (DEFAULT_TABLE table=test-table [id=ca5b5be25bdc4f7cadf23cd5bd82bca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:02:18.702518 11563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1f422bb8f43498fbd80e808668a4eea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:18.704882 11656 tablet_bootstrap.cc:492] T d1f422bb8f43498fbd80e808668a4eea P 20c06efe6f604539900f0458b56bb974: Bootstrap starting.
I20260806 01:02:18.705641 11656 tablet_bootstrap.cc:654] T d1f422bb8f43498fbd80e808668a4eea P 20c06efe6f604539900f0458b56bb974: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:18.705533 11290 tablet_service.cc:1467] Processing CreateTablet for tablet d1f422bb8f43498fbd80e808668a4eea (DEFAULT_TABLE table=test-table [id=ca5b5be25bdc4f7cadf23cd5bd82bca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:02:18.705917 11290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1f422bb8f43498fbd80e808668a4eea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:18.706025 11656 log.cc:824] T d1f422bb8f43498fbd80e808668a4eea P 20c06efe6f604539900f0458b56bb974: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:18.706801 11658 tablet_bootstrap.cc:492] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2: Bootstrap starting.
I20260806 01:02:18.706970 11656 tablet_bootstrap.cc:492] T d1f422bb8f43498fbd80e808668a4eea P 20c06efe6f604539900f0458b56bb974: No bootstrap required, opened a new log
I20260806 01:02:18.707053 11656 ts_tablet_manager.cc:1397] T d1f422bb8f43498fbd80e808668a4eea P 20c06efe6f604539900f0458b56bb974: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 01:02:18.707958 11658 tablet_bootstrap.cc:654] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:18.708501 11658 log.cc:824] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:18.709611 11658 tablet_bootstrap.cc:492] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2: No bootstrap required, opened a new log
I20260806 01:02:18.709746 11658 ts_tablet_manager.cc:1397] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 01:02:18.710016 11660 tablet_bootstrap.cc:492] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5: Bootstrap starting.
I20260806 01:02:18.710261 11656 raft_consensus.cc:348] T d1f422bb8f43498fbd80e808668a4eea P 20c06efe6f604539900f0458b56bb974 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793e3e9f04084db8ac69dbe3473515d2" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 38203 } } peers { permanent_uuid: "20c06efe6f604539900f0458b56bb974" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38257 } } peers { permanent_uuid: "23c299bf9cd34d5fbe8376cb40dd9fe5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44045 } }
I20260806 01:02:18.710474 11656 raft_consensus.cc:374] T d1f422bb8f43498fbd80e808668a4eea P 20c06efe6f604539900f0458b56bb974 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:18.710513 11656 raft_consensus.cc:729] T d1f422bb8f43498fbd80e808668a4eea P 20c06efe6f604539900f0458b56bb974 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20c06efe6f604539900f0458b56bb974, State: Initialized, Role: FOLLOWER
I20260806 01:02:18.710770 11660 tablet_bootstrap.cc:654] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:18.710767 11656 consensus_queue.cc:260] T d1f422bb8f43498fbd80e808668a4eea P 20c06efe6f604539900f0458b56bb974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793e3e9f04084db8ac69dbe3473515d2" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 38203 } } peers { permanent_uuid: "20c06efe6f604539900f0458b56bb974" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38257 } } peers { permanent_uuid: "23c299bf9cd34d5fbe8376cb40dd9fe5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44045 } }
I20260806 01:02:18.711100 11656 ts_tablet_manager.cc:1428] T d1f422bb8f43498fbd80e808668a4eea P 20c06efe6f604539900f0458b56bb974: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260806 01:02:18.711179 11496 heartbeater.cc:503] Master 127.23.242.62:37409 was elected leader, sending a full tablet report...
I20260806 01:02:18.711192 11660 log.cc:824] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:18.712181 11660 tablet_bootstrap.cc:492] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5: No bootstrap required, opened a new log
I20260806 01:02:18.712272 11660 ts_tablet_manager.cc:1397] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260806 01:02:18.713043 11658 raft_consensus.cc:348] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793e3e9f04084db8ac69dbe3473515d2" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 38203 } } peers { permanent_uuid: "20c06efe6f604539900f0458b56bb974" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38257 } } peers { permanent_uuid: "23c299bf9cd34d5fbe8376cb40dd9fe5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44045 } }
I20260806 01:02:18.713267 11658 raft_consensus.cc:374] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:18.713315 11658 raft_consensus.cc:729] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793e3e9f04084db8ac69dbe3473515d2, State: Initialized, Role: FOLLOWER
I20260806 01:02:18.713442 11658 consensus_queue.cc:260] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793e3e9f04084db8ac69dbe3473515d2" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 38203 } } peers { permanent_uuid: "20c06efe6f604539900f0458b56bb974" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38257 } } peers { permanent_uuid: "23c299bf9cd34d5fbe8376cb40dd9fe5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44045 } }
I20260806 01:02:18.713770 11658 ts_tablet_manager.cc:1428] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260806 01:02:18.713910 11638 heartbeater.cc:503] Master 127.23.242.62:37409 was elected leader, sending a full tablet report...
I20260806 01:02:18.715240 11660 raft_consensus.cc:348] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793e3e9f04084db8ac69dbe3473515d2" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 38203 } } peers { permanent_uuid: "20c06efe6f604539900f0458b56bb974" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38257 } } peers { permanent_uuid: "23c299bf9cd34d5fbe8376cb40dd9fe5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44045 } }
I20260806 01:02:18.715404 11660 raft_consensus.cc:374] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:18.715448 11660 raft_consensus.cc:729] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c299bf9cd34d5fbe8376cb40dd9fe5, State: Initialized, Role: FOLLOWER
I20260806 01:02:18.715591 11660 consensus_queue.cc:260] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793e3e9f04084db8ac69dbe3473515d2" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 38203 } } peers { permanent_uuid: "20c06efe6f604539900f0458b56bb974" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38257 } } peers { permanent_uuid: "23c299bf9cd34d5fbe8376cb40dd9fe5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44045 } }
I20260806 01:02:18.715868 11660 ts_tablet_manager.cc:1428] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260806 01:02:18.716012 11359 heartbeater.cc:503] Master 127.23.242.62:37409 was elected leader, sending a full tablet report...
I20260806 01:02:18.752389 11664 raft_consensus.cc:482] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 01:02:18.752542 11664 raft_consensus.cc:504] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793e3e9f04084db8ac69dbe3473515d2" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 38203 } } peers { permanent_uuid: "20c06efe6f604539900f0458b56bb974" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38257 } } peers { permanent_uuid: "23c299bf9cd34d5fbe8376cb40dd9fe5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44045 } }
I20260806 01:02:18.753111 11664 leader_election.cc:288] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793e3e9f04084db8ac69dbe3473515d2 (127.23.242.3:38203), 20c06efe6f604539900f0458b56bb974 (127.23.242.2:38257)
I20260806 01:02:18.753223 11663 raft_consensus.cc:482] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 01:02:18.753378 11663 raft_consensus.cc:504] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793e3e9f04084db8ac69dbe3473515d2" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 38203 } } peers { permanent_uuid: "20c06efe6f604539900f0458b56bb974" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38257 } } peers { permanent_uuid: "23c299bf9cd34d5fbe8376cb40dd9fe5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44045 } }
I20260806 01:02:18.753746 11663 leader_election.cc:288] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20c06efe6f604539900f0458b56bb974 (127.23.242.2:38257), 23c299bf9cd34d5fbe8376cb40dd9fe5 (127.23.242.1:44045)
W20260806 01:02:18.754233 11497 tablet.cc:2369] T d1f422bb8f43498fbd80e808668a4eea P 20c06efe6f604539900f0458b56bb974: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 01:02:18.756958 11361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:18.753337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:38203 (local address 127.23.242.1:38579)
0806 01:02:18.753403 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:18.753415 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:18.753464 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:18.753697 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:18.753698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:18.753700 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:18.753832 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:18.753835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.754785 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.754786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:18.756021 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:18.756024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.756701 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.756702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:18.756781 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:18.756785 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 01:02:18.756836 (+ 51us) client_negotiation.cc:240] Negotiation successful
0806 01:02:18.756877 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260806 01:02:18.757413 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:18.753467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:38579 (local address 127.23.242.3:38203)
0806 01:02:18.753542 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:18.753545 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:18.753555 (+ 10us) server_negotiation.cc:407] Connection header received
0806 01:02:18.753585 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:18.753586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:18.753596 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:18.753644 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:18.753926 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.754689 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.756103 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.756610 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.756703 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:18.756712 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:18.757317 (+ 605us) server_negotiation.cc:299] Negotiation successful
0806 01:02:18.757353 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 01:02:18.757797 11654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:18.753832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:46883 (local address 127.23.242.2:38257)
0806 01:02:18.753909 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:18.753911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:18.753921 (+ 10us) server_negotiation.cc:407] Connection header received
0806 01:02:18.753951 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:18.753952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:18.753962 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:18.754017 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:18.754286 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.755056 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.756819 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.757409 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.757454 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:18.757461 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:18.757688 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 01:02:18.757726 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 01:02:18.757776 11587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f422bb8f43498fbd80e808668a4eea" candidate_uuid: "23c299bf9cd34d5fbe8376cb40dd9fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793e3e9f04084db8ac69dbe3473515d2" is_pre_election: true
I20260806 01:02:18.757879 11587 raft_consensus.cc:2454] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23c299bf9cd34d5fbe8376cb40dd9fe5 in term 0.
I20260806 01:02:18.758064 11665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:18.753351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:38257 (local address 127.23.242.1:46883)
0806 01:02:18.753570 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:18.753582 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:18.753641 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:18.754104 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:18.754105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:18.754107 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:18.754187 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:18.754188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.755179 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.755180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:18.756707 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:18.756711 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.757524 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.757525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:18.757603 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:18.757606 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 01:02:18.757916 (+ 310us) client_negotiation.cc:240] Negotiation successful
0806 01:02:18.757972 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":132,"threads_started":1}
I20260806 01:02:18.758297 11640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:18.753877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:38257 (local address 127.23.242.3:52307)
0806 01:02:18.754102 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:18.754112 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:18.754151 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:18.754748 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:18.754749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:18.754751 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:18.754815 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:18.754816 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.755764 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.755765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:18.757168 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:18.757171 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.758039 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.758040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:18.758123 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:18.758127 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 01:02:18.758167 (+ 40us) client_negotiation.cc:240] Negotiation successful
0806 01:02:18.758219 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":170}
I20260806 01:02:18.758404 11242 leader_election.cc:302] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23c299bf9cd34d5fbe8376cb40dd9fe5, 793e3e9f04084db8ac69dbe3473515d2; no voters:
I20260806 01:02:18.758397 11449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f422bb8f43498fbd80e808668a4eea" candidate_uuid: "23c299bf9cd34d5fbe8376cb40dd9fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20c06efe6f604539900f0458b56bb974" is_pre_election: true
I20260806 01:02:18.758559 11669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:18.753990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:52307 (local address 127.23.242.2:38257)
0806 01:02:18.754616 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:18.754617 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:18.754624 (+ 7us) server_negotiation.cc:407] Connection header received
0806 01:02:18.754644 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:18.754644 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:18.754653 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:18.754876 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:18.754908 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.755661 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.757265 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.757924 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.758411 (+ 487us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:18.758420 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:18.758441 (+ 21us) server_negotiation.cc:299] Negotiation successful
0806 01:02:18.758479 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":97,"threads_started":1}
I20260806 01:02:18.758659 11653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:18.754266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:37695 (local address 127.23.242.1:44045)
0806 01:02:18.754495 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:18.754497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:18.754506 (+ 9us) server_negotiation.cc:407] Connection header received
0806 01:02:18.754526 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:18.754527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:18.754537 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:18.754630 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:18.754835 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.755611 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.757659 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.758211 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.758253 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:18.758260 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:18.758491 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 01:02:18.758527 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":207}
I20260806 01:02:18.758539 11664 raft_consensus.cc:2781] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 01:02:18.758760 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:18.753956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:44045 (local address 127.23.242.3:37695)
0806 01:02:18.754357 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:18.754367 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:18.754407 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:18.754677 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:18.754678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:18.754681 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:18.754753 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:18.754755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.755711 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.755712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:18.757561 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:18.757564 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.758335 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.758335 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:18.758403 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:18.758406 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 01:02:18.758553 (+ 147us) client_negotiation.cc:240] Negotiation successful
0806 01:02:18.758636 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":70,"thread_start_us":105,"threads_started":1}
I20260806 01:02:18.758541 11449 raft_consensus.cc:2454] T d1f422bb8f43498fbd80e808668a4eea P 20c06efe6f604539900f0458b56bb974 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23c299bf9cd34d5fbe8376cb40dd9fe5 in term 0.
I20260806 01:02:18.758805 11664 raft_consensus.cc:482] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 01:02:18.758844 11664 raft_consensus.cc:3037] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:18.759111 11449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f422bb8f43498fbd80e808668a4eea" candidate_uuid: "793e3e9f04084db8ac69dbe3473515d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20c06efe6f604539900f0458b56bb974" is_pre_election: true
I20260806 01:02:18.759127 11310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f422bb8f43498fbd80e808668a4eea" candidate_uuid: "793e3e9f04084db8ac69dbe3473515d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23c299bf9cd34d5fbe8376cb40dd9fe5" is_pre_election: true
I20260806 01:02:18.759182 11449 raft_consensus.cc:2454] T d1f422bb8f43498fbd80e808668a4eea P 20c06efe6f604539900f0458b56bb974 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793e3e9f04084db8ac69dbe3473515d2 in term 0.
I20260806 01:02:18.759399 11517 leader_election.cc:302] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20c06efe6f604539900f0458b56bb974, 793e3e9f04084db8ac69dbe3473515d2; no voters:
I20260806 01:02:18.759517 11663 raft_consensus.cc:2781] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 01:02:18.759562 11663 raft_consensus.cc:482] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 01:02:18.759588 11663 raft_consensus.cc:3037] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:18.759899 11664 raft_consensus.cc:504] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793e3e9f04084db8ac69dbe3473515d2" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 38203 } } peers { permanent_uuid: "20c06efe6f604539900f0458b56bb974" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38257 } } peers { permanent_uuid: "23c299bf9cd34d5fbe8376cb40dd9fe5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44045 } }
I20260806 01:02:18.760063 11310 raft_consensus.cc:2379] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 793e3e9f04084db8ac69dbe3473515d2 in current term 1: Already voted for candidate 23c299bf9cd34d5fbe8376cb40dd9fe5 in this term.
I20260806 01:02:18.760215 11664 leader_election.cc:288] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5 [CANDIDATE]: Term 1 election: Requested vote from peers 793e3e9f04084db8ac69dbe3473515d2 (127.23.242.3:38203), 20c06efe6f604539900f0458b56bb974 (127.23.242.2:38257)
I20260806 01:02:18.760216 11587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f422bb8f43498fbd80e808668a4eea" candidate_uuid: "23c299bf9cd34d5fbe8376cb40dd9fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793e3e9f04084db8ac69dbe3473515d2"
I20260806 01:02:18.760380 11449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f422bb8f43498fbd80e808668a4eea" candidate_uuid: "23c299bf9cd34d5fbe8376cb40dd9fe5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20c06efe6f604539900f0458b56bb974"
I20260806 01:02:18.760462 11449 raft_consensus.cc:3037] T d1f422bb8f43498fbd80e808668a4eea P 20c06efe6f604539900f0458b56bb974 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:18.760473 11663 raft_consensus.cc:504] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793e3e9f04084db8ac69dbe3473515d2" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 38203 } } peers { permanent_uuid: "20c06efe6f604539900f0458b56bb974" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38257 } } peers { permanent_uuid: "23c299bf9cd34d5fbe8376cb40dd9fe5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44045 } }
I20260806 01:02:18.760596 11587 raft_consensus.cc:2379] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 23c299bf9cd34d5fbe8376cb40dd9fe5 in current term 1: Already voted for candidate 793e3e9f04084db8ac69dbe3473515d2 in this term.
I20260806 01:02:18.760630 11663 leader_election.cc:288] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2 [CANDIDATE]: Term 1 election: Requested vote from peers 20c06efe6f604539900f0458b56bb974 (127.23.242.2:38257), 23c299bf9cd34d5fbe8376cb40dd9fe5 (127.23.242.1:44045)
I20260806 01:02:18.760762 11447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f422bb8f43498fbd80e808668a4eea" candidate_uuid: "793e3e9f04084db8ac69dbe3473515d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20c06efe6f604539900f0458b56bb974"
I20260806 01:02:18.760830 11310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f422bb8f43498fbd80e808668a4eea" candidate_uuid: "793e3e9f04084db8ac69dbe3473515d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23c299bf9cd34d5fbe8376cb40dd9fe5"
I20260806 01:02:18.760913 11310 raft_consensus.cc:2379] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 793e3e9f04084db8ac69dbe3473515d2 in current term 1: Already voted for candidate 23c299bf9cd34d5fbe8376cb40dd9fe5 in this term.
I20260806 01:02:18.761097 11515 leader_election.cc:302] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 793e3e9f04084db8ac69dbe3473515d2; no voters: 20c06efe6f604539900f0458b56bb974, 23c299bf9cd34d5fbe8376cb40dd9fe5
I20260806 01:02:18.761204 11663 raft_consensus.cc:2726] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 01:02:18.761452 11449 raft_consensus.cc:2454] T d1f422bb8f43498fbd80e808668a4eea P 20c06efe6f604539900f0458b56bb974 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23c299bf9cd34d5fbe8376cb40dd9fe5 in term 1.
I20260806 01:02:18.761910 11245 leader_election.cc:302] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20c06efe6f604539900f0458b56bb974, 23c299bf9cd34d5fbe8376cb40dd9fe5; no voters: 793e3e9f04084db8ac69dbe3473515d2
I20260806 01:02:18.762041 11664 raft_consensus.cc:2781] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:18.762215 11664 raft_consensus.cc:686] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5 [term 1 LEADER]: Becoming Leader. State: Replica: 23c299bf9cd34d5fbe8376cb40dd9fe5, State: Running, Role: LEADER
I20260806 01:02:18.762298 11664 consensus_queue.cc:237] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793e3e9f04084db8ac69dbe3473515d2" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 38203 } } peers { permanent_uuid: "20c06efe6f604539900f0458b56bb974" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38257 } } peers { permanent_uuid: "23c299bf9cd34d5fbe8376cb40dd9fe5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44045 } }
I20260806 01:02:18.763939 11163 catalog_manager.cc:5495] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23c299bf9cd34d5fbe8376cb40dd9fe5 (127.23.242.1). New cstate: current_term: 1 leader_uuid: "23c299bf9cd34d5fbe8376cb40dd9fe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793e3e9f04084db8ac69dbe3473515d2" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 38203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20c06efe6f604539900f0458b56bb974" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23c299bf9cd34d5fbe8376cb40dd9fe5" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 44045 } health_report { overall_health: HEALTHY } } }
I20260806 01:02:18.777864 11653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:18.774685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55458 (local address 127.23.242.1:44045)
0806 01:02:18.774757 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:18.774759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:18.774769 (+ 10us) server_negotiation.cc:407] Connection header received
0806 01:02:18.774815 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:18.774816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:18.774843 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:18.774926 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 01:02:18.775107 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.775704 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.776905 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:18.777130 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:18.777219 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:18.777451 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 01:02:18.777552 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 01:02:18.777690 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:18.777736 (+ 46us) server_negotiation.cc:299] Negotiation successful
0806 01:02:18.777751 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 01:02:18.780012 11449 raft_consensus.cc:1261] T d1f422bb8f43498fbd80e808668a4eea P 20c06efe6f604539900f0458b56bb974 [term 1 FOLLOWER]: Refusing update from remote peer 23c299bf9cd34d5fbe8376cb40dd9fe5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:02:18.780062 11587 raft_consensus.cc:1261] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2 [term 1 FOLLOWER]: Refusing update from remote peer 23c299bf9cd34d5fbe8376cb40dd9fe5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:02:18.780411 11670 consensus_queue.cc:1035] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793e3e9f04084db8ac69dbe3473515d2" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:02:18.780607 11664 consensus_queue.cc:1035] T d1f422bb8f43498fbd80e808668a4eea P 23c299bf9cd34d5fbe8376cb40dd9fe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20c06efe6f604539900f0458b56bb974" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 38257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:02:18.783313 11674 mvcc.cc:205] Tried to move back new op lower bound from 7315366456440025088 to 7315366456371081216. Current Snapshot: MvccSnapshot[applied={T|T < 7315366456440025088}]
I20260806 01:02:18.786405 11163 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 01:02:18.786532 11163 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 01:02:18.787586 11163 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e56fb3454f9c405d8c27bd6bd2b9638d: Sending DeleteTablet for 3 replicas of tablet d1f422bb8f43498fbd80e808668a4eea
I20260806 01:02:18.787984 11563 tablet_service.cc:1514] Processing DeleteTablet for tablet d1f422bb8f43498fbd80e808668a4eea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:02:18 UTC) from {username='slave'} at 127.0.0.1:51990
I20260806 01:02:18.788203 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 11227
I20260806 01:02:18.788326 11428 tablet_service.cc:1514] Processing DeleteTablet for tablet d1f422bb8f43498fbd80e808668a4eea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:02:18 UTC) from {username='slave'} at 127.0.0.1:33314
I20260806 01:02:18.789572 11683 tablet_replica.cc:330] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2: stopping tablet replica
I20260806 01:02:18.789752 11683 raft_consensus.cc:2229] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 01:02:18.789898 11683 raft_consensus.cc:2258] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 01:02:18.790393 11684 tablet_replica.cc:330] T d1f422bb8f43498fbd80e808668a4eea P 20c06efe6f604539900f0458b56bb974: stopping tablet replica
I20260806 01:02:18.790527 11684 raft_consensus.cc:2229] T d1f422bb8f43498fbd80e808668a4eea P 20c06efe6f604539900f0458b56bb974 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 01:02:18.790685 11684 raft_consensus.cc:2258] T d1f422bb8f43498fbd80e808668a4eea P 20c06efe6f604539900f0458b56bb974 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 01:02:18.790710 11683 ts_tablet_manager.cc:1905] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 01:02:18.791267 11684 ts_tablet_manager.cc:1905] T d1f422bb8f43498fbd80e808668a4eea P 20c06efe6f604539900f0458b56bb974: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 01:02:18.793133 11683 ts_tablet_manager.cc:1918] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 01:02:18.793287 11683 log.cc:1196] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-2/wal/wals/d1f422bb8f43498fbd80e808668a4eea
I20260806 01:02:18.793697 11684 ts_tablet_manager.cc:1918] T d1f422bb8f43498fbd80e808668a4eea P 20c06efe6f604539900f0458b56bb974: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 01:02:18.793726 11683 ts_tablet_manager.cc:1939] T d1f422bb8f43498fbd80e808668a4eea P 793e3e9f04084db8ac69dbe3473515d2: Deleting consensus metadata
I20260806 01:02:18.793797 11684 log.cc:1196] T d1f422bb8f43498fbd80e808668a4eea P 20c06efe6f604539900f0458b56bb974: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785977869987768-24520-0/minicluster-data/ts-1/wal/wals/d1f422bb8f43498fbd80e808668a4eea
I20260806 01:02:18.794117 11684 ts_tablet_manager.cc:1939] T d1f422bb8f43498fbd80e808668a4eea P 20c06efe6f604539900f0458b56bb974: Deleting consensus metadata
I20260806 01:02:18.794479 11148 catalog_manager.cc:4841] TS 793e3e9f04084db8ac69dbe3473515d2 (127.23.242.3:38203): tablet d1f422bb8f43498fbd80e808668a4eea (table test-table [id=ca5b5be25bdc4f7cadf23cd5bd82bca5]) successfully deleted
I20260806 01:02:18.794641 11151 catalog_manager.cc:4841] TS 20c06efe6f604539900f0458b56bb974 (127.23.242.2:38257): tablet d1f422bb8f43498fbd80e808668a4eea (table test-table [id=ca5b5be25bdc4f7cadf23cd5bd82bca5]) successfully deleted
W20260806 01:02:18.796363 11149 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.242.1:44045 (error 108)
I20260806 01:02:18.796696 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 11362
I20260806 01:02:18.797870 11650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:18.797693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:44045 (local address 127.0.0.1:55472)
0806 01:02:18.797757 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:18.797821 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.1:44045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260806 01:02:18.798028 11149 catalog_manager.cc:4584] TS 23c299bf9cd34d5fbe8376cb40dd9fe5 (127.23.242.1:44045): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d1f422bb8f43498fbd80e808668a4eea: Network error: Client connection negotiation failed: client connection to 127.23.242.1:44045: connect: Connection refused (error 111)
I20260806 01:02:18.804455 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 11499
I20260806 01:02:18.811466 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 11130
2026-08-06T01:02:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (849 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 01:02:18 dist-test-slave-dhph krb5kdc[11689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 01:02:18 dist-test-slave-dhph krb5kdc[11689](info): set up 2 sockets
Aug 06 01:02:18 dist-test-slave-dhph krb5kdc[11689](info): commencing operation
krb5kdc: starting...
W20260806 01:02:20.041334 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.208s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 01:02:20 dist-test-slave-dhph krb5kdc[11689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T01:02:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:02:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 01:02:20.187505 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:39621
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:39621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:20.287420 11705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:20.287690 11705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:20.287732 11705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:20.289824 11705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:20.289932 11705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:20.289961 11705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:20.289985 11705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:20.290016 11705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:20.292508 11705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:39621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:39621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:20.292986 11705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:20.293426 11705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:20.300004 11711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:20.300005 11710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:20.300004 11713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:20.300035 11705 server_base.cc:1034] running on GCE node
I20260806 01:02:20.300654 11705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:20.300962 11705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:20.302220 11705 hybrid_clock.cc:648] HybridClock initialized: now 1785978140302185 us; error 50 us; skew 500 ppm
Aug 06 01:02:20 dist-test-slave-dhph krb5kdc[11689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978140, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:20.315903 11705 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 01:02:20.316502 11705 webserver.cc:460] Webserver started at http://127.23.242.62:44967/ using document root <none> and password file <none>
I20260806 01:02:20.316757 11705 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:20.316833 11705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:20.316989 11705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:20.318455 11705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "792f22ac90534ff89cdec7f7e7bb63fb"
format_stamp: "Formatted at 2026-08-06 01:02:20 on dist-test-slave-dhph"
server_key: "dca120fac279ac5e971c3293e4b7d4a7"
server_key_iv: "6bf3e954fa5e4919731156b4159c22ea"
server_key_version: "encryptionkey@0"
I20260806 01:02:20.318928 11705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "792f22ac90534ff89cdec7f7e7bb63fb"
format_stamp: "Formatted at 2026-08-06 01:02:20 on dist-test-slave-dhph"
server_key: "dca120fac279ac5e971c3293e4b7d4a7"
server_key_iv: "6bf3e954fa5e4919731156b4159c22ea"
server_key_version: "encryptionkey@0"
I20260806 01:02:20.320533 11705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:20.321700 11721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:20.322022 11705 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:20.322161 11705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "792f22ac90534ff89cdec7f7e7bb63fb"
format_stamp: "Formatted at 2026-08-06 01:02:20 on dist-test-slave-dhph"
server_key: "dca120fac279ac5e971c3293e4b7d4a7"
server_key_iv: "6bf3e954fa5e4919731156b4159c22ea"
server_key_version: "encryptionkey@0"
I20260806 01:02:20.322252 11705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:20.341109 11705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:20.344254 11705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:20.344455 11705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:20.349282 11705 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:39621
I20260806 01:02:20.349336 11778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:39621 every 8 connection(s)
I20260806 01:02:20.349767 11705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:02:20.350847 11779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:20.353509 11779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 792f22ac90534ff89cdec7f7e7bb63fb: Bootstrap starting.
I20260806 01:02:20.354158 11779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 792f22ac90534ff89cdec7f7e7bb63fb: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:20.354424 11779 log.cc:824] T 00000000000000000000000000000000 P 792f22ac90534ff89cdec7f7e7bb63fb: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:20.355113 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 11705
I20260806 01:02:20.355245 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:02:20.355414 11779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 792f22ac90534ff89cdec7f7e7bb63fb: No bootstrap required, opened a new log
I20260806 01:02:20.355437 24520 external_mini_cluster.cc:1428] Setting key f68b0ad0e8538674bd3618b9ce9dfe8d
I20260806 01:02:20.357656 11779 raft_consensus.cc:348] T 00000000000000000000000000000000 P 792f22ac90534ff89cdec7f7e7bb63fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792f22ac90534ff89cdec7f7e7bb63fb" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39621 } }
I20260806 01:02:20.358034 11779 raft_consensus.cc:374] T 00000000000000000000000000000000 P 792f22ac90534ff89cdec7f7e7bb63fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:20.358144 11779 raft_consensus.cc:729] T 00000000000000000000000000000000 P 792f22ac90534ff89cdec7f7e7bb63fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 792f22ac90534ff89cdec7f7e7bb63fb, State: Initialized, Role: FOLLOWER
I20260806 01:02:20.358336 11779 consensus_queue.cc:260] T 00000000000000000000000000000000 P 792f22ac90534ff89cdec7f7e7bb63fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792f22ac90534ff89cdec7f7e7bb63fb" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39621 } }
I20260806 01:02:20.358449 11779 raft_consensus.cc:388] T 00000000000000000000000000000000 P 792f22ac90534ff89cdec7f7e7bb63fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:02:20.358491 11779 raft_consensus.cc:482] T 00000000000000000000000000000000 P 792f22ac90534ff89cdec7f7e7bb63fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:02:20.358534 11779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 792f22ac90534ff89cdec7f7e7bb63fb [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:20.359540 11779 raft_consensus.cc:504] T 00000000000000000000000000000000 P 792f22ac90534ff89cdec7f7e7bb63fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792f22ac90534ff89cdec7f7e7bb63fb" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39621 } }
I20260806 01:02:20.359670 11779 leader_election.cc:302] T 00000000000000000000000000000000 P 792f22ac90534ff89cdec7f7e7bb63fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 792f22ac90534ff89cdec7f7e7bb63fb; no voters:
I20260806 01:02:20.359895 11779 leader_election.cc:288] T 00000000000000000000000000000000 P 792f22ac90534ff89cdec7f7e7bb63fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:02:20.359935 11784 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 792f22ac90534ff89cdec7f7e7bb63fb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:20.360046 11784 raft_consensus.cc:686] T 00000000000000000000000000000000 P 792f22ac90534ff89cdec7f7e7bb63fb [term 1 LEADER]: Becoming Leader. State: Replica: 792f22ac90534ff89cdec7f7e7bb63fb, State: Running, Role: LEADER
I20260806 01:02:20.360134 11784 consensus_queue.cc:237] T 00000000000000000000000000000000 P 792f22ac90534ff89cdec7f7e7bb63fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792f22ac90534ff89cdec7f7e7bb63fb" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39621 } }
I20260806 01:02:20.360199 11779 sys_catalog.cc:564] T 00000000000000000000000000000000 P 792f22ac90534ff89cdec7f7e7bb63fb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:20.360463 11784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 792f22ac90534ff89cdec7f7e7bb63fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 792f22ac90534ff89cdec7f7e7bb63fb. Latest consensus state: current_term: 1 leader_uuid: "792f22ac90534ff89cdec7f7e7bb63fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792f22ac90534ff89cdec7f7e7bb63fb" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39621 } } }
I20260806 01:02:20.360539 11784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 792f22ac90534ff89cdec7f7e7bb63fb [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:20.360486 11785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 792f22ac90534ff89cdec7f7e7bb63fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "792f22ac90534ff89cdec7f7e7bb63fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "792f22ac90534ff89cdec7f7e7bb63fb" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 39621 } } }
I20260806 01:02:20.360713 11785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 792f22ac90534ff89cdec7f7e7bb63fb [sys.catalog]: This master's current role is: LEADER
W20260806 01:02:20.362195 11799 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 792f22ac90534ff89cdec7f7e7bb63fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 01:02:20.362308 11799 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 01:02:20.362586 11788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:02:20.362819 11788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:02:20.364470 11788 catalog_manager.cc:1347] Generated new cluster ID: 7f996c3eabc0486a96894f1d03f4db0e
I20260806 01:02:20.364528 11788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 01:02:20 dist-test-slave-dhph krb5kdc[11689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:02:20.371829 11783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:20.356002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57042 (local address 127.23.242.62:39621)
0806 01:02:20.356227 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:20.356235 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:20.356264 (+ 29us) server_negotiation.cc:407] Connection header received
0806 01:02:20.357724 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:20.357741 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:20.358313 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:20.358525 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:20.359315 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:20.360620 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:20.361615 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:20.362271 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:20.362778 (+ 507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:20.366316 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:20.366322 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:20.366322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:20.366336 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:20.368743 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:20.369404 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:20.369413 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:20.369414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:20.369656 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:20.370097 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:20.370098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:20.370099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:20.370958 (+ 859us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:20.371206 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:20.371286 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 01:02:20.371583 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":123,"threads_started":1}
I20260806 01:02:20.381758 11788 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:02:20.382566 11788 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:02:20.394652 11788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 792f22ac90534ff89cdec7f7e7bb63fb: Generated new TSK 0
I20260806 01:02:20.394920 11788 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.242.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
Entry for principal kudu/127.23.242.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab.
I20260806 01:02:20.448587 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:0
--local_ip_for_outbound_sockets=127.23.242.1
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:39621
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:20.568943 11805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:20.569235 11805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:20.569278 11805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:20.570658 11805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:20.570744 11805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:20.570799 11805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 01:02:20.572532 11805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.1.keytab
--principal=kudu/127.23.242.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:39621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:20.572852 11805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:20.573133 11805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:20.580319 11811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:20.580319 11812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:20.580438 11814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:20.580648 11805 server_base.cc:1034] running on GCE node
I20260806 01:02:20.580881 11805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:20.581183 11805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:20.582420 11805 hybrid_clock.cc:648] HybridClock initialized: now 1785978140582382 us; error 56 us; skew 500 ppm
Aug 06 01:02:20 dist-test-slave-dhph krb5kdc[11689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978140, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:20.585675 11805 init.cc:377] Logged in from keytab as kudu/127.23.242.1@KRBTEST.COM (short username kudu)
I20260806 01:02:20.586189 11805 webserver.cc:460] Webserver started at http://127.23.242.1:32891/ using document root <none> and password file <none>
I20260806 01:02:20.586470 11805 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:20.586550 11805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:20.586774 11805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:20.588074 11805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-0/data/instance:
uuid: "23b67f77e2044f7da9eb351b32360566"
format_stamp: "Formatted at 2026-08-06 01:02:20 on dist-test-slave-dhph"
server_key: "362bc91a2c70aa103a6b0fe2c319d070"
server_key_iv: "1a4e4f584f356e3b531081fbb359480e"
server_key_version: "encryptionkey@0"
I20260806 01:02:20.588497 11805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance:
uuid: "23b67f77e2044f7da9eb351b32360566"
format_stamp: "Formatted at 2026-08-06 01:02:20 on dist-test-slave-dhph"
server_key: "362bc91a2c70aa103a6b0fe2c319d070"
server_key_iv: "1a4e4f584f356e3b531081fbb359480e"
server_key_version: "encryptionkey@0"
I20260806 01:02:20.590116 11805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 01:02:20.591199 11824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:20.591558 11805 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 01:02:20.591688 11805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "23b67f77e2044f7da9eb351b32360566"
format_stamp: "Formatted at 2026-08-06 01:02:20 on dist-test-slave-dhph"
server_key: "362bc91a2c70aa103a6b0fe2c319d070"
server_key_iv: "1a4e4f584f356e3b531081fbb359480e"
server_key_version: "encryptionkey@0"
I20260806 01:02:20.591804 11805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:20.608448 11805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:20.611426 11805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:20.611635 11805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:20.611999 11805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:02:20.612360 11805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:02:20.612424 11805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:20.612471 11805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:02:20.612502 11805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:20.621730 11805 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:46489
I20260806 01:02:20.621857 11944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:46489 every 8 connection(s)
I20260806 01:02:20.622329 11805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 01:02:20.628235 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 11805
I20260806 01:02:20.628404 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance
I20260806 01:02:20.628676 24520 external_mini_cluster.cc:1428] Setting key 1c01e330065a803a104125c8e933fa5a
Aug 06 01:02:20 dist-test-slave-dhph krb5kdc[11689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978140, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.1@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:02:20.636181 11783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:20.623624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:54293 (local address 127.23.242.62:39621)
0806 01:02:20.623731 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:20.623734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:20.623937 (+ 203us) server_negotiation.cc:407] Connection header received
0806 01:02:20.624577 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:20.624579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:20.624600 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:20.624668 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:20.625837 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:20.626479 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:20.627764 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:20.628060 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:20.628236 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:20.631804 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:20.631806 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:20.631807 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:20.631815 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:20.634369 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:20.634989 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:20.634990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:20.634991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:20.635051 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:20.635365 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:20.635366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:20.635366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:20.635505 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:20.635591 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:20.635919 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 01:02:20.636029 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 01:02:20.636431 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:20.623474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:39621 (local address 127.23.242.1:54293)
0806 01:02:20.623774 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:20.623834 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:20.624426 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:20.624773 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:20.624783 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:20.625471 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:20.625706 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:20.625711 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:20.626653 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:20.626654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:20.627596 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:20.627599 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:20.628255 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:20.628257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:20.628462 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:20.629666 (+ 1204us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:02:20.629675 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:20.631495 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:20.634476 (+ 2981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:20.634489 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:20.634492 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:20.634882 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:20.635135 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:20.635136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:20.635136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:20.635279 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:20.635580 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:20.635588 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:20.635762 (+ 174us) client_negotiation.cc:769] Sending connection context
0806 01:02:20.635880 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 01:02:20.636092 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":126,"threads_started":1}
I20260806 01:02:20.636981 11945 heartbeater.cc:347] Connected to a master server at 127.23.242.62:39621
I20260806 01:02:20.637091 11945 heartbeater.cc:464] Registering TS with master...
I20260806 01:02:20.637332 11945 heartbeater.cc:511] Master 127.23.242.62:39621 requested a full tablet report, sending...
I20260806 01:02:20.638417 11740 ts_manager.cc:194] Registered new tserver with Master: 23b67f77e2044f7da9eb351b32360566 (127.23.242.1:46489)
I20260806 01:02:20.639397 11740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.1@KRBTEST.COM'} at 127.23.242.1:54293
WARNING: no policy specified for kudu/127.23.242.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
Entry for principal kudu/127.23.242.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab.
I20260806 01:02:20.668905 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:0
--local_ip_for_outbound_sockets=127.23.242.2
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:39621
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:20.769055 11950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:20.769302 11950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:20.769333 11950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:20.770555 11950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:20.770622 11950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:20.770694 11950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 01:02:20.772162 11950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.2.keytab
--principal=kudu/127.23.242.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:39621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:20.772367 11950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:20.772559 11950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:20.778083 11957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:20.778074 11959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:20.778242 11950 server_base.cc:1034] running on GCE node
W20260806 01:02:20.778074 11956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:20.778741 11950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:20.779062 11950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:20.780259 11950 hybrid_clock.cc:648] HybridClock initialized: now 1785978140780220 us; error 46 us; skew 500 ppm
Aug 06 01:02:20 dist-test-slave-dhph krb5kdc[11689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978140, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:20.783463 11950 init.cc:377] Logged in from keytab as kudu/127.23.242.2@KRBTEST.COM (short username kudu)
I20260806 01:02:20.784173 11950 webserver.cc:460] Webserver started at http://127.23.242.2:45157/ using document root <none> and password file <none>
I20260806 01:02:20.784459 11950 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:20.784513 11950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:20.784621 11950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:20.785848 11950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-1/data/instance:
uuid: "eee89e86c87f4c3cabb2fe6edd411165"
format_stamp: "Formatted at 2026-08-06 01:02:20 on dist-test-slave-dhph"
server_key: "b14242064273b557b948dd5adbbd24d6"
server_key_iv: "73be45f07cb4e63f3f9ef97d4a4a49f1"
server_key_version: "encryptionkey@0"
I20260806 01:02:20.786240 11950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance:
uuid: "eee89e86c87f4c3cabb2fe6edd411165"
format_stamp: "Formatted at 2026-08-06 01:02:20 on dist-test-slave-dhph"
server_key: "b14242064273b557b948dd5adbbd24d6"
server_key_iv: "73be45f07cb4e63f3f9ef97d4a4a49f1"
server_key_version: "encryptionkey@0"
I20260806 01:02:20.787914 11950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 01:02:20.788825 11967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:20.789157 11950 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:20.789280 11950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "eee89e86c87f4c3cabb2fe6edd411165"
format_stamp: "Formatted at 2026-08-06 01:02:20 on dist-test-slave-dhph"
server_key: "b14242064273b557b948dd5adbbd24d6"
server_key_iv: "73be45f07cb4e63f3f9ef97d4a4a49f1"
server_key_version: "encryptionkey@0"
I20260806 01:02:20.789362 11950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:20.811215 11950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:20.813822 11950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:20.814023 11950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:20.814322 11950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:02:20.814713 11950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:02:20.814754 11950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:20.814779 11950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:02:20.814832 11950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:20.822642 11950 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:42519
I20260806 01:02:20.822816 12086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:42519 every 8 connection(s)
I20260806 01:02:20.823123 11950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 01:02:20.826740 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 11950
I20260806 01:02:20.827031 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance
I20260806 01:02:20.827364 24520 external_mini_cluster.cc:1428] Setting key 9b68682c68599f7d9362f770f1970efc
Aug 06 01:02:20 dist-test-slave-dhph krb5kdc[11689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978140, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.2@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:02:20.836511 11783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:20.824189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:48953 (local address 127.23.242.62:39621)
0806 01:02:20.824288 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:20.824290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:20.824709 (+ 419us) server_negotiation.cc:407] Connection header received
0806 01:02:20.825303 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:20.825304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:20.825318 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:20.825386 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:20.826711 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:20.827431 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:20.828881 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:20.829222 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:20.829280 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:20.832689 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:20.832692 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:20.832693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:20.832704 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:20.834788 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:20.835367 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:20.835369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:20.835369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:20.835438 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:20.835768 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:20.835769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:20.835769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:20.835909 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:20.835997 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:20.836301 (+ 304us) server_negotiation.cc:299] Negotiation successful
0806 01:02:20.836398 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 01:02:20.836762 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:20.824209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:39621 (local address 127.23.242.2:48953)
0806 01:02:20.824418 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:20.824459 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:20.825181 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:20.825468 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:20.825472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:20.826226 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:20.826531 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:20.826537 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:20.827581 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:20.827582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:20.828732 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:20.828735 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:20.829404 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:20.829405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:20.829604 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:20.830331 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:02:20.830340 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:20.832484 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:20.834925 (+ 2441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:20.834939 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:20.834941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:20.835257 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:20.835521 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:20.835522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:20.835522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:20.835649 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:20.835988 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:20.835996 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:20.836150 (+ 154us) client_negotiation.cc:769] Sending connection context
0806 01:02:20.836278 (+ 128us) client_negotiation.cc:240] Negotiation successful
0806 01:02:20.836478 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":102,"threads_started":1}
I20260806 01:02:20.837774 12087 heartbeater.cc:347] Connected to a master server at 127.23.242.62:39621
I20260806 01:02:20.838023 12087 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.242.3@KRBTEST.COM; defaulting to no policy
I20260806 01:02:20.838435 12087 heartbeater.cc:511] Master 127.23.242.62:39621 requested a full tablet report, sending...
I20260806 01:02:20.839172 11740 ts_manager.cc:194] Registered new tserver with Master: eee89e86c87f4c3cabb2fe6edd411165 (127.23.242.2:42519)
I20260806 01:02:20.839710 11740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.2@KRBTEST.COM'} at 127.23.242.2:48953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
Entry for principal kudu/127.23.242.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab.
I20260806 01:02:20.863734 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:0
--local_ip_for_outbound_sockets=127.23.242.3
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:39621
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:20.963032 12092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:20.963287 12092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:20.963318 12092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:20.964622 12092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:20.964682 12092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:20.964715 12092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 01:02:20.966327 12092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.3.keytab
--principal=kudu/127.23.242.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:39621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:20.966648 12092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:20.966925 12092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:20.972812 12100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:20.972812 12097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:20.972846 12098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:20.973407 12092 server_base.cc:1034] running on GCE node
I20260806 01:02:20.973609 12092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:20.973884 12092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:20.975060 12092 hybrid_clock.cc:648] HybridClock initialized: now 1785978140975046 us; error 30 us; skew 500 ppm
Aug 06 01:02:20 dist-test-slave-dhph krb5kdc[11689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978140, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:20.977922 12092 init.cc:377] Logged in from keytab as kudu/127.23.242.3@KRBTEST.COM (short username kudu)
I20260806 01:02:20.978466 12092 webserver.cc:460] Webserver started at http://127.23.242.3:41897/ using document root <none> and password file <none>
I20260806 01:02:20.978785 12092 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:20.978873 12092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:20.979043 12092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:20.980279 12092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-2/data/instance:
uuid: "e2dbddbfe2004716b023be2d02274a7c"
format_stamp: "Formatted at 2026-08-06 01:02:20 on dist-test-slave-dhph"
server_key: "9fe96a3102943f1b1c4cf185bb1d359b"
server_key_iv: "aa1afeb324b5dea69f5c41cf3523602d"
server_key_version: "encryptionkey@0"
I20260806 01:02:20.980676 12092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance:
uuid: "e2dbddbfe2004716b023be2d02274a7c"
format_stamp: "Formatted at 2026-08-06 01:02:20 on dist-test-slave-dhph"
server_key: "9fe96a3102943f1b1c4cf185bb1d359b"
server_key_iv: "aa1afeb324b5dea69f5c41cf3523602d"
server_key_version: "encryptionkey@0"
I20260806 01:02:20.982190 12092 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 01:02:20.983194 12109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:20.983510 12092 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 01:02:20.983662 12092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "e2dbddbfe2004716b023be2d02274a7c"
format_stamp: "Formatted at 2026-08-06 01:02:20 on dist-test-slave-dhph"
server_key: "9fe96a3102943f1b1c4cf185bb1d359b"
server_key_iv: "aa1afeb324b5dea69f5c41cf3523602d"
server_key_version: "encryptionkey@0"
I20260806 01:02:20.983796 12092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:20.997033 12092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:21.000208 12092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:21.000543 12092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:21.000844 12092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:02:21.001248 12092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:02:21.001313 12092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:21.001362 12092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:02:21.001410 12092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:21.009577 12092 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:37137
I20260806 01:02:21.009622 12230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:37137 every 8 connection(s)
I20260806 01:02:21.009997 12092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
I20260806 01:02:21.010557 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 12092
I20260806 01:02:21.010684 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance
I20260806 01:02:21.010901 24520 external_mini_cluster.cc:1428] Setting key b5c3401b28be15313666dbaf91371fb1
Aug 06 01:02:21 dist-test-slave-dhph krb5kdc[11689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978140, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.3@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:02:21.022359 11783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.010856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:55173 (local address 127.23.242.62:39621)
0806 01:02:21.010932 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:21.010935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:21.011413 (+ 478us) server_negotiation.cc:407] Connection header received
0806 01:02:21.012030 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:21.012032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:21.012045 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:21.012174 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:21.013231 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.013810 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.015084 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.015448 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.015546 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.018749 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:21.018750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:21.018750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:21.018759 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:21.020474 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:21.021082 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:21.021084 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:21.021084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:21.021154 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:21.021478 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:21.021479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:21.021479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:21.021628 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:21.021695 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:21.022192 (+ 497us) server_negotiation.cc:299] Negotiation successful
0806 01:02:21.022280 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 01:02:21.022714 12233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.010838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:39621 (local address 127.23.242.3:55173)
0806 01:02:21.011077 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:21.011121 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:21.011831 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:21.012263 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:21.012268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:21.012909 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:21.013112 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.013116 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.013924 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.013925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.014918 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.014921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.015621 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.015624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.015815 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.016643 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:02:21.016654 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:21.018562 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:21.020621 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:21.020634 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:21.020637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:21.020952 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:21.021257 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:21.021258 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:21.021258 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:21.021385 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:21.021738 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:21.021744 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:21.021969 (+ 225us) client_negotiation.cc:769] Sending connection context
0806 01:02:21.022119 (+ 150us) client_negotiation.cc:240] Negotiation successful
0806 01:02:21.022287 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":93,"threads_started":1}
I20260806 01:02:21.023345 12231 heartbeater.cc:347] Connected to a master server at 127.23.242.62:39621
I20260806 01:02:21.023473 12231 heartbeater.cc:464] Registering TS with master...
I20260806 01:02:21.023716 12231 heartbeater.cc:511] Master 127.23.242.62:39621 requested a full tablet report, sending...
I20260806 01:02:21.024225 11740 ts_manager.cc:194] Registered new tserver with Master: e2dbddbfe2004716b023be2d02274a7c (127.23.242.3:37137)
I20260806 01:02:21.024801 11740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.242.3@KRBTEST.COM'} at 127.23.242.3:55173
I20260806 01:02:21.034982 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 01:02:21.043938 11783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.036718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57046 (local address 127.23.242.62:39621)
0806 01:02:21.036818 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:21.036820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:21.036846 (+ 26us) server_negotiation.cc:407] Connection header received
0806 01:02:21.036900 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:21.036901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:21.036912 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:21.036958 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:21.037628 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.038241 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.039121 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.039322 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.039387 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.040784 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:21.040786 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:21.040786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:21.040793 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:21.042316 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:21.042803 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:21.042804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:21.042805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:21.042868 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:21.043098 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:21.043099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:21.043099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:21.043316 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:21.043410 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:21.043701 (+ 291us) server_negotiation.cc:299] Negotiation successful
0806 01:02:21.043792 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 01:02:21.046278 11740 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 01:02:21.048264 11740 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 01:02:21.056802 12242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.051646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:37137 (local address 127.0.0.1:33480)
0806 01:02:21.052128 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:21.052324 (+ 196us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:21.052523 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:21.053178 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:21.053181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:21.053185 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:21.053281 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.053284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.054413 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.054413 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.055657 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.055660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.056430 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.056431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.056536 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.056555 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 01:02:21.056639 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 01:02:21.056703 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":259,"threads_started":1}
I20260806 01:02:21.056842 12246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.051963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33480 (local address 127.23.242.3:37137)
0806 01:02:21.052337 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:21.052341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:21.052455 (+ 114us) server_negotiation.cc:407] Connection header received
0806 01:02:21.052682 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:21.052686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:21.052709 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:21.052783 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:21.053399 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.054288 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.055771 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.056325 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.056427 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.056500 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:21.056692 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 01:02:21.056749 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":124,"threads_started":1}
I20260806 01:02:21.057971 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.052492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41020 (local address 127.23.242.1:46489)
0806 01:02:21.053163 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:21.053170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:21.053182 (+ 12us) server_negotiation.cc:407] Connection header received
0806 01:02:21.053224 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:21.053226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:21.053245 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:21.053344 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:21.053589 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.054910 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.056564 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.057445 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.057518 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.057587 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:21.057829 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 01:02:21.057883 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":140,"threads_started":1}
I20260806 01:02:21.057986 12243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.051718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:42519 (local address 127.0.0.1:51038)
0806 01:02:21.052411 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:21.052423 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:21.052614 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:21.053214 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:21.053216 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:21.053219 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:21.053307 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.053310 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.055060 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.055062 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.056451 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.056455 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.057644 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.057645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.057719 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.057726 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 01:02:21.057790 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 01:02:21.057845 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":62,"thread_start_us":65,"threads_started":1}
I20260806 01:02:21.057994 12245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.051953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:46489 (local address 127.0.0.1:41020)
0806 01:02:21.052755 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:21.052767 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:21.052962 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:21.053457 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:21.053459 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:21.053462 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:21.053511 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.053513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.055060 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.055062 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.056468 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.056470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.057639 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.057640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.057719 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.057727 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 01:02:21.057791 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 01:02:21.057845 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":42,"thread_start_us":94,"threads_started":1}
I20260806 01:02:21.058127 12159 tablet_service.cc:1467] Processing CreateTablet for tablet e3e9adb00b4241f8b381a1d6b98d7d27 (DEFAULT_TABLE table=test-table [id=2a9849fa279345148739f1da70d41a0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:02:21.058480 12244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.051882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51038 (local address 127.23.242.2:42519)
0806 01:02:21.052181 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:21.052190 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:21.052855 (+ 665us) server_negotiation.cc:407] Connection header received
0806 01:02:21.053010 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:21.053019 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:21.053085 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:21.053214 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:21.053516 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.054917 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.056573 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.057467 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.057558 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.057643 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:21.058338 (+ 695us) server_negotiation.cc:299] Negotiation successful
0806 01:02:21.058391 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":128,"threads_started":1}
I20260806 01:02:21.058566 12159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3e9adb00b4241f8b381a1d6b98d7d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:21.059175 11875 tablet_service.cc:1467] Processing CreateTablet for tablet e3e9adb00b4241f8b381a1d6b98d7d27 (DEFAULT_TABLE table=test-table [id=2a9849fa279345148739f1da70d41a0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:02:21.059321 12017 tablet_service.cc:1467] Processing CreateTablet for tablet e3e9adb00b4241f8b381a1d6b98d7d27 (DEFAULT_TABLE table=test-table [id=2a9849fa279345148739f1da70d41a0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:02:21.059548 11875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3e9adb00b4241f8b381a1d6b98d7d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:21.059648 12017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3e9adb00b4241f8b381a1d6b98d7d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:21.063292 12251 tablet_bootstrap.cc:492] T e3e9adb00b4241f8b381a1d6b98d7d27 P eee89e86c87f4c3cabb2fe6edd411165: Bootstrap starting.
I20260806 01:02:21.063391 12249 tablet_bootstrap.cc:492] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c: Bootstrap starting.
I20260806 01:02:21.064184 12251 tablet_bootstrap.cc:654] T e3e9adb00b4241f8b381a1d6b98d7d27 P eee89e86c87f4c3cabb2fe6edd411165: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:21.064195 12249 tablet_bootstrap.cc:654] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:21.064739 12251 log.cc:824] T e3e9adb00b4241f8b381a1d6b98d7d27 P eee89e86c87f4c3cabb2fe6edd411165: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:21.065074 12249 log.cc:824] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:21.065131 12250 tablet_bootstrap.cc:492] T e3e9adb00b4241f8b381a1d6b98d7d27 P 23b67f77e2044f7da9eb351b32360566: Bootstrap starting.
I20260806 01:02:21.065546 12251 tablet_bootstrap.cc:492] T e3e9adb00b4241f8b381a1d6b98d7d27 P eee89e86c87f4c3cabb2fe6edd411165: No bootstrap required, opened a new log
I20260806 01:02:21.065644 12251 ts_tablet_manager.cc:1397] T e3e9adb00b4241f8b381a1d6b98d7d27 P eee89e86c87f4c3cabb2fe6edd411165: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 01:02:21.065920 12249 tablet_bootstrap.cc:492] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c: No bootstrap required, opened a new log
I20260806 01:02:21.065981 12249 ts_tablet_manager.cc:1397] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 01:02:21.067299 12251 raft_consensus.cc:348] T e3e9adb00b4241f8b381a1d6b98d7d27 P eee89e86c87f4c3cabb2fe6edd411165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dbddbfe2004716b023be2d02274a7c" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 37137 } } peers { permanent_uuid: "eee89e86c87f4c3cabb2fe6edd411165" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 42519 } } peers { permanent_uuid: "23b67f77e2044f7da9eb351b32360566" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46489 } }
I20260806 01:02:21.067505 12251 raft_consensus.cc:374] T e3e9adb00b4241f8b381a1d6b98d7d27 P eee89e86c87f4c3cabb2fe6edd411165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:21.067595 12251 raft_consensus.cc:729] T e3e9adb00b4241f8b381a1d6b98d7d27 P eee89e86c87f4c3cabb2fe6edd411165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eee89e86c87f4c3cabb2fe6edd411165, State: Initialized, Role: FOLLOWER
I20260806 01:02:21.067777 12251 consensus_queue.cc:260] T e3e9adb00b4241f8b381a1d6b98d7d27 P eee89e86c87f4c3cabb2fe6edd411165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dbddbfe2004716b023be2d02274a7c" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 37137 } } peers { permanent_uuid: "eee89e86c87f4c3cabb2fe6edd411165" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 42519 } } peers { permanent_uuid: "23b67f77e2044f7da9eb351b32360566" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46489 } }
I20260806 01:02:21.068166 12251 ts_tablet_manager.cc:1428] T e3e9adb00b4241f8b381a1d6b98d7d27 P eee89e86c87f4c3cabb2fe6edd411165: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 01:02:21.068291 12087 heartbeater.cc:503] Master 127.23.242.62:39621 was elected leader, sending a full tablet report...
I20260806 01:02:21.068563 12249 raft_consensus.cc:348] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dbddbfe2004716b023be2d02274a7c" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 37137 } } peers { permanent_uuid: "eee89e86c87f4c3cabb2fe6edd411165" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 42519 } } peers { permanent_uuid: "23b67f77e2044f7da9eb351b32360566" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46489 } }
I20260806 01:02:21.068957 12249 raft_consensus.cc:374] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:21.069209 12250 tablet_bootstrap.cc:654] T e3e9adb00b4241f8b381a1d6b98d7d27 P 23b67f77e2044f7da9eb351b32360566: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:21.069247 12249 raft_consensus.cc:729] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2dbddbfe2004716b023be2d02274a7c, State: Initialized, Role: FOLLOWER
I20260806 01:02:21.069475 12249 consensus_queue.cc:260] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dbddbfe2004716b023be2d02274a7c" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 37137 } } peers { permanent_uuid: "eee89e86c87f4c3cabb2fe6edd411165" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 42519 } } peers { permanent_uuid: "23b67f77e2044f7da9eb351b32360566" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46489 } }
I20260806 01:02:21.069880 12249 ts_tablet_manager.cc:1428] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260806 01:02:21.070006 12231 heartbeater.cc:503] Master 127.23.242.62:39621 was elected leader, sending a full tablet report...
I20260806 01:02:21.070015 12250 log.cc:824] T e3e9adb00b4241f8b381a1d6b98d7d27 P 23b67f77e2044f7da9eb351b32360566: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:21.070916 12250 tablet_bootstrap.cc:492] T e3e9adb00b4241f8b381a1d6b98d7d27 P 23b67f77e2044f7da9eb351b32360566: No bootstrap required, opened a new log
I20260806 01:02:21.071014 12250 ts_tablet_manager.cc:1397] T e3e9adb00b4241f8b381a1d6b98d7d27 P 23b67f77e2044f7da9eb351b32360566: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260806 01:02:21.073238 12250 raft_consensus.cc:348] T e3e9adb00b4241f8b381a1d6b98d7d27 P 23b67f77e2044f7da9eb351b32360566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dbddbfe2004716b023be2d02274a7c" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 37137 } } peers { permanent_uuid: "eee89e86c87f4c3cabb2fe6edd411165" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 42519 } } peers { permanent_uuid: "23b67f77e2044f7da9eb351b32360566" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46489 } }
I20260806 01:02:21.073454 12250 raft_consensus.cc:374] T e3e9adb00b4241f8b381a1d6b98d7d27 P 23b67f77e2044f7da9eb351b32360566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:21.073516 12250 raft_consensus.cc:729] T e3e9adb00b4241f8b381a1d6b98d7d27 P 23b67f77e2044f7da9eb351b32360566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b67f77e2044f7da9eb351b32360566, State: Initialized, Role: FOLLOWER
I20260806 01:02:21.073666 12250 consensus_queue.cc:260] T e3e9adb00b4241f8b381a1d6b98d7d27 P 23b67f77e2044f7da9eb351b32360566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dbddbfe2004716b023be2d02274a7c" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 37137 } } peers { permanent_uuid: "eee89e86c87f4c3cabb2fe6edd411165" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 42519 } } peers { permanent_uuid: "23b67f77e2044f7da9eb351b32360566" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46489 } }
W20260806 01:02:21.073828 12088 tablet.cc:2369] T e3e9adb00b4241f8b381a1d6b98d7d27 P eee89e86c87f4c3cabb2fe6edd411165: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 01:02:21.074038 11945 heartbeater.cc:503] Master 127.23.242.62:39621 was elected leader, sending a full tablet report...
I20260806 01:02:21.074041 12250 ts_tablet_manager.cc:1428] T e3e9adb00b4241f8b381a1d6b98d7d27 P 23b67f77e2044f7da9eb351b32360566: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 01:02:21.090298 12255 raft_consensus.cc:482] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 01:02:21.090489 12255 raft_consensus.cc:504] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dbddbfe2004716b023be2d02274a7c" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 37137 } } peers { permanent_uuid: "eee89e86c87f4c3cabb2fe6edd411165" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 42519 } } peers { permanent_uuid: "23b67f77e2044f7da9eb351b32360566" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46489 } }
I20260806 01:02:21.091298 12255 leader_election.cc:288] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eee89e86c87f4c3cabb2fe6edd411165 (127.23.242.2:42519), 23b67f77e2044f7da9eb351b32360566 (127.23.242.1:46489)
I20260806 01:02:21.095675 12244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.091619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:34655 (local address 127.23.242.2:42519)
0806 01:02:21.091678 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:21.091680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:21.091692 (+ 12us) server_negotiation.cc:407] Connection header received
0806 01:02:21.091726 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:21.091727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:21.091739 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:21.091840 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:21.092029 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.092959 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.094749 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.095233 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.095336 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.095347 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:21.095525 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 01:02:21.095567 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 01:02:21.095712 12233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.091460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:42519 (local address 127.23.242.3:34655)
0806 01:02:21.091541 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:21.091554 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:21.091616 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:21.091864 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:21.091865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:21.091867 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:21.091948 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.091950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.093092 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.093094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.094627 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.094630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.095332 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.095333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.095429 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.095439 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 01:02:21.095516 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 01:02:21.095567 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":1}
I20260806 01:02:21.095686 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.091713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:49749 (local address 127.23.242.1:46489)
0806 01:02:21.091792 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:21.091795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:21.091914 (+ 119us) server_negotiation.cc:407] Connection header received
0806 01:02:21.091973 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:21.091974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:21.091982 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:21.092057 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:21.092373 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.093140 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.094850 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.095251 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.095373 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.095382 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:21.095565 (+ 183us) server_negotiation.cc:299] Negotiation successful
0806 01:02:21.095610 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 01:02:21.095726 12258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.091604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:46489 (local address 127.23.242.3:49749)
0806 01:02:21.091831 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:21.091840 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:21.091914 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:21.092059 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:21.092060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:21.092061 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:21.092140 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.092142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.093245 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.093246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.094627 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.094630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.095380 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.095381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.095469 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.095473 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 01:02:21.095556 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 01:02:21.095599 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":152,"threads_started":1}
I20260806 01:02:21.096114 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e9adb00b4241f8b381a1d6b98d7d27" candidate_uuid: "e2dbddbfe2004716b023be2d02274a7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eee89e86c87f4c3cabb2fe6edd411165" is_pre_election: true
I20260806 01:02:21.096304 12039 raft_consensus.cc:2454] T e3e9adb00b4241f8b381a1d6b98d7d27 P eee89e86c87f4c3cabb2fe6edd411165 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2dbddbfe2004716b023be2d02274a7c in term 0.
I20260806 01:02:21.096386 11895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e9adb00b4241f8b381a1d6b98d7d27" candidate_uuid: "e2dbddbfe2004716b023be2d02274a7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b67f77e2044f7da9eb351b32360566" is_pre_election: true
I20260806 01:02:21.096534 11895 raft_consensus.cc:2454] T e3e9adb00b4241f8b381a1d6b98d7d27 P 23b67f77e2044f7da9eb351b32360566 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2dbddbfe2004716b023be2d02274a7c in term 0.
I20260806 01:02:21.096560 12114 leader_election.cc:302] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e2dbddbfe2004716b023be2d02274a7c, eee89e86c87f4c3cabb2fe6edd411165; no voters:
I20260806 01:02:21.096712 12255 raft_consensus.cc:2781] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 01:02:21.096776 12255 raft_consensus.cc:482] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 01:02:21.096817 12255 raft_consensus.cc:3037] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:21.097606 12255 raft_consensus.cc:504] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dbddbfe2004716b023be2d02274a7c" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 37137 } } peers { permanent_uuid: "eee89e86c87f4c3cabb2fe6edd411165" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 42519 } } peers { permanent_uuid: "23b67f77e2044f7da9eb351b32360566" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46489 } }
I20260806 01:02:21.097764 12255 leader_election.cc:288] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c [CANDIDATE]: Term 1 election: Requested vote from peers eee89e86c87f4c3cabb2fe6edd411165 (127.23.242.2:42519), 23b67f77e2044f7da9eb351b32360566 (127.23.242.1:46489)
I20260806 01:02:21.097939 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e9adb00b4241f8b381a1d6b98d7d27" candidate_uuid: "e2dbddbfe2004716b023be2d02274a7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eee89e86c87f4c3cabb2fe6edd411165"
I20260806 01:02:21.097949 11895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3e9adb00b4241f8b381a1d6b98d7d27" candidate_uuid: "e2dbddbfe2004716b023be2d02274a7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b67f77e2044f7da9eb351b32360566"
I20260806 01:02:21.098012 12039 raft_consensus.cc:3037] T e3e9adb00b4241f8b381a1d6b98d7d27 P eee89e86c87f4c3cabb2fe6edd411165 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:21.098062 11895 raft_consensus.cc:3037] T e3e9adb00b4241f8b381a1d6b98d7d27 P 23b67f77e2044f7da9eb351b32360566 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:21.099151 11895 raft_consensus.cc:2454] T e3e9adb00b4241f8b381a1d6b98d7d27 P 23b67f77e2044f7da9eb351b32360566 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dbddbfe2004716b023be2d02274a7c in term 1.
I20260806 01:02:21.099359 12114 leader_election.cc:302] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b67f77e2044f7da9eb351b32360566, e2dbddbfe2004716b023be2d02274a7c; no voters:
I20260806 01:02:21.099572 12039 raft_consensus.cc:2454] T e3e9adb00b4241f8b381a1d6b98d7d27 P eee89e86c87f4c3cabb2fe6edd411165 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2dbddbfe2004716b023be2d02274a7c in term 1.
I20260806 01:02:21.102591 12255 raft_consensus.cc:2781] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:21.102849 12255 raft_consensus.cc:686] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c [term 1 LEADER]: Becoming Leader. State: Replica: e2dbddbfe2004716b023be2d02274a7c, State: Running, Role: LEADER
I20260806 01:02:21.102968 12255 consensus_queue.cc:237] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dbddbfe2004716b023be2d02274a7c" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 37137 } } peers { permanent_uuid: "eee89e86c87f4c3cabb2fe6edd411165" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 42519 } } peers { permanent_uuid: "23b67f77e2044f7da9eb351b32360566" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46489 } }
I20260806 01:02:21.106345 11739 catalog_manager.cc:5495] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c reported cstate change: term changed from 0 to 1, leader changed from <none> to e2dbddbfe2004716b023be2d02274a7c (127.23.242.3). New cstate: current_term: 1 leader_uuid: "e2dbddbfe2004716b023be2d02274a7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2dbddbfe2004716b023be2d02274a7c" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 37137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eee89e86c87f4c3cabb2fe6edd411165" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 42519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23b67f77e2044f7da9eb351b32360566" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46489 } health_report { overall_health: UNKNOWN } } }
W20260806 01:02:21.123610 11946 tablet.cc:2369] T e3e9adb00b4241f8b381a1d6b98d7d27 P 23b67f77e2044f7da9eb351b32360566: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 01:02:21.125998 12246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.121763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33488 (local address 127.23.242.3:37137)
0806 01:02:21.121818 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:21.121820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:21.121861 (+ 41us) server_negotiation.cc:407] Connection header received
0806 01:02:21.121974 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:21.121975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:21.122003 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:21.122107 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 01:02:21.122230 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.123147 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.124564 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.124935 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.125030 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.125348 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 01:02:21.125503 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 01:02:21.125626 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:21.125811 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 01:02:21.125842 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260806 01:02:21.128301 12039 raft_consensus.cc:1261] T e3e9adb00b4241f8b381a1d6b98d7d27 P eee89e86c87f4c3cabb2fe6edd411165 [term 1 FOLLOWER]: Refusing update from remote peer e2dbddbfe2004716b023be2d02274a7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:02:21.128388 11895 raft_consensus.cc:1261] T e3e9adb00b4241f8b381a1d6b98d7d27 P 23b67f77e2044f7da9eb351b32360566 [term 1 FOLLOWER]: Refusing update from remote peer e2dbddbfe2004716b023be2d02274a7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:02:21.128777 12259 consensus_queue.cc:1035] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b67f77e2044f7da9eb351b32360566" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 46489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:02:21.128934 12255 consensus_queue.cc:1035] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eee89e86c87f4c3cabb2fe6edd411165" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 42519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:02:21.138087 11740 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 01:02:21.138275 11740 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 01:02:21.139442 11740 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 792f22ac90534ff89cdec7f7e7bb63fb: Sending DeleteTablet for 3 replicas of tablet e3e9adb00b4241f8b381a1d6b98d7d27
I20260806 01:02:21.139971 12159 tablet_service.cc:1514] Processing DeleteTablet for tablet e3e9adb00b4241f8b381a1d6b98d7d27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:02:21 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:33480
I20260806 01:02:21.140012 12017 tablet_service.cc:1514] Processing DeleteTablet for tablet e3e9adb00b4241f8b381a1d6b98d7d27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:02:21 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:51038
I20260806 01:02:21.140255 11875 tablet_service.cc:1514] Processing DeleteTablet for tablet e3e9adb00b4241f8b381a1d6b98d7d27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:02:21 UTC) from {username='kudu', principal='kudu/127.23.242.62@KRBTEST.COM'} at 127.0.0.1:41020
I20260806 01:02:21.140468 12274 tablet_replica.cc:330] T e3e9adb00b4241f8b381a1d6b98d7d27 P 23b67f77e2044f7da9eb351b32360566: stopping tablet replica
I20260806 01:02:21.140628 12274 raft_consensus.cc:2229] T e3e9adb00b4241f8b381a1d6b98d7d27 P 23b67f77e2044f7da9eb351b32360566 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 01:02:21.140869 12273 tablet_replica.cc:330] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c: stopping tablet replica
I20260806 01:02:21.140923 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 11805
I20260806 01:02:21.141043 12273 raft_consensus.cc:2229] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c [term 1 LEADER]: Raft consensus shutting down.
I20260806 01:02:21.141186 12273 raft_consensus.cc:2258] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 01:02:21.144544 12273 ts_tablet_manager.cc:1905] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 01:02:21.147333 12273 ts_tablet_manager.cc:1918] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 01:02:21.147435 12273 log.cc:1196] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-2/wal/wals/e3e9adb00b4241f8b381a1d6b98d7d27
I20260806 01:02:21.147830 12273 ts_tablet_manager.cc:1939] T e3e9adb00b4241f8b381a1d6b98d7d27 P e2dbddbfe2004716b023be2d02274a7c: Deleting consensus metadata
I20260806 01:02:21.148965 11724 catalog_manager.cc:4841] TS e2dbddbfe2004716b023be2d02274a7c (127.23.242.3:37137): tablet e3e9adb00b4241f8b381a1d6b98d7d27 (table test-table [id=2a9849fa279345148739f1da70d41a0f]) successfully deleted
I20260806 01:02:21.150769 12272 tablet_replica.cc:330] T e3e9adb00b4241f8b381a1d6b98d7d27 P eee89e86c87f4c3cabb2fe6edd411165: stopping tablet replica
I20260806 01:02:21.151008 12272 raft_consensus.cc:2229] T e3e9adb00b4241f8b381a1d6b98d7d27 P eee89e86c87f4c3cabb2fe6edd411165 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 01:02:21.151224 12272 raft_consensus.cc:2258] T e3e9adb00b4241f8b381a1d6b98d7d27 P eee89e86c87f4c3cabb2fe6edd411165 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 01:02:21.152154 12272 ts_tablet_manager.cc:1905] T e3e9adb00b4241f8b381a1d6b98d7d27 P eee89e86c87f4c3cabb2fe6edd411165: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 01:02:21.155529 12272 ts_tablet_manager.cc:1918] T e3e9adb00b4241f8b381a1d6b98d7d27 P eee89e86c87f4c3cabb2fe6edd411165: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 01:02:21.155728 12272 log.cc:1196] T e3e9adb00b4241f8b381a1d6b98d7d27 P eee89e86c87f4c3cabb2fe6edd411165: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785977869987768-24520-0/minicluster-data/ts-1/wal/wals/e3e9adb00b4241f8b381a1d6b98d7d27
I20260806 01:02:21.156267 12272 ts_tablet_manager.cc:1939] T e3e9adb00b4241f8b381a1d6b98d7d27 P eee89e86c87f4c3cabb2fe6edd411165: Deleting consensus metadata
I20260806 01:02:21.157224 11726 catalog_manager.cc:4841] TS eee89e86c87f4c3cabb2fe6edd411165 (127.23.242.2:42519): tablet e3e9adb00b4241f8b381a1d6b98d7d27 (table test-table [id=2a9849fa279345148739f1da70d41a0f]) successfully deleted
W20260806 01:02:21.157990 11726 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.242.1:46489 (error 108)
I20260806 01:02:21.158397 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 11950
I20260806 01:02:21.167181 12245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.166994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:46489 (local address 127.0.0.1:41036)
0806 01:02:21.167064 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:21.167111 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.1:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 01:02:21.167506 11726 catalog_manager.cc:4584] TS 23b67f77e2044f7da9eb351b32360566 (127.23.242.1:46489): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e3e9adb00b4241f8b381a1d6b98d7d27: Network error: Client connection negotiation failed: client connection to 127.23.242.1:46489: connect: Connection refused (error 111)
I20260806 01:02:21.168444 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 12092
I20260806 01:02:21.175683 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 11705
2026-08-06T01:02:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2368 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-06T01:02:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:02:21Z Disabled control of system clock
I20260806 01:02:21.212342 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:36595
--webserver_interface=127.23.242.62
--webserver_port=0
--builtin_ntp_servers=127.23.242.20:43695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:36595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 01:02:21.310789 12279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:21.311054 12279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:21.311077 12279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:21.312274 12279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:21.312326 12279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:21.312340 12279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:21.312353 12279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:21.312366 12279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:21.313779 12279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:43695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:36595
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:36595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:21.314039 12279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:21.314354 12279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:21.319875 12287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:21.319880 12285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:21.319893 12289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:21.320295 12279 server_base.cc:1034] running on GCE node
I20260806 01:02:21.320586 12279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:21.320899 12279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:21.322160 12279 hybrid_clock.cc:648] HybridClock initialized: now 1785978141322117 us; error 56 us; skew 500 ppm
I20260806 01:02:21.323901 12279 webserver.cc:460] Webserver started at http://127.23.242.62:34233/ using document root <none> and password file <none>
I20260806 01:02:21.324223 12279 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:21.324307 12279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:21.324481 12279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:21.325742 12279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "3ad519ddc3504a51a833d5fff84f523c"
format_stamp: "Formatted at 2026-08-06 01:02:21 on dist-test-slave-dhph"
server_key: "cd2b39abdfa7b1b1a84675107f82a4fb"
server_key_iv: "ad8c5650bf355042b218a55a2a85f80d"
server_key_version: "encryptionkey@0"
I20260806 01:02:21.326465 12279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "3ad519ddc3504a51a833d5fff84f523c"
format_stamp: "Formatted at 2026-08-06 01:02:21 on dist-test-slave-dhph"
server_key: "cd2b39abdfa7b1b1a84675107f82a4fb"
server_key_iv: "ad8c5650bf355042b218a55a2a85f80d"
server_key_version: "encryptionkey@0"
I20260806 01:02:21.328090 12279 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:21.329142 12295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:21.329385 12279 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:21.329514 12279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "3ad519ddc3504a51a833d5fff84f523c"
format_stamp: "Formatted at 2026-08-06 01:02:21 on dist-test-slave-dhph"
server_key: "cd2b39abdfa7b1b1a84675107f82a4fb"
server_key_iv: "ad8c5650bf355042b218a55a2a85f80d"
server_key_version: "encryptionkey@0"
I20260806 01:02:21.329605 12279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:21.355501 12279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:21.355922 12279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:21.356141 12279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:21.362418 12279 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:36595
I20260806 01:02:21.362458 12350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:36595 every 8 connection(s)
I20260806 01:02:21.362982 12279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:02:21.364277 12351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:21.366797 12351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ad519ddc3504a51a833d5fff84f523c: Bootstrap starting.
I20260806 01:02:21.367460 12351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ad519ddc3504a51a833d5fff84f523c: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:21.367815 12351 log.cc:824] T 00000000000000000000000000000000 P 3ad519ddc3504a51a833d5fff84f523c: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:21.368557 12351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ad519ddc3504a51a833d5fff84f523c: No bootstrap required, opened a new log
I20260806 01:02:21.369279 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 12279
I20260806 01:02:21.369436 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:02:21.369670 24520 external_mini_cluster.cc:1428] Setting key e7011381f58d9b9b826c5f3a55a88ed1
I20260806 01:02:21.371493 12351 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ad519ddc3504a51a833d5fff84f523c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ad519ddc3504a51a833d5fff84f523c" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 36595 } }
I20260806 01:02:21.371711 12351 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ad519ddc3504a51a833d5fff84f523c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:21.371759 12351 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ad519ddc3504a51a833d5fff84f523c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ad519ddc3504a51a833d5fff84f523c, State: Initialized, Role: FOLLOWER
I20260806 01:02:21.371907 12351 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ad519ddc3504a51a833d5fff84f523c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ad519ddc3504a51a833d5fff84f523c" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 36595 } }
I20260806 01:02:21.372010 12351 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ad519ddc3504a51a833d5fff84f523c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:02:21.372049 12351 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ad519ddc3504a51a833d5fff84f523c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:02:21.372094 12351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ad519ddc3504a51a833d5fff84f523c [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:21.373075 12351 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ad519ddc3504a51a833d5fff84f523c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ad519ddc3504a51a833d5fff84f523c" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 36595 } }
I20260806 01:02:21.373210 12351 leader_election.cc:302] T 00000000000000000000000000000000 P 3ad519ddc3504a51a833d5fff84f523c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ad519ddc3504a51a833d5fff84f523c; no voters:
I20260806 01:02:21.373430 12351 leader_election.cc:288] T 00000000000000000000000000000000 P 3ad519ddc3504a51a833d5fff84f523c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:02:21.373560 12356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ad519ddc3504a51a833d5fff84f523c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:21.373714 12351 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ad519ddc3504a51a833d5fff84f523c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:21.373831 12356 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ad519ddc3504a51a833d5fff84f523c [term 1 LEADER]: Becoming Leader. State: Replica: 3ad519ddc3504a51a833d5fff84f523c, State: Running, Role: LEADER
I20260806 01:02:21.373930 12356 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ad519ddc3504a51a833d5fff84f523c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ad519ddc3504a51a833d5fff84f523c" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 36595 } }
I20260806 01:02:21.374354 12357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ad519ddc3504a51a833d5fff84f523c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ad519ddc3504a51a833d5fff84f523c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ad519ddc3504a51a833d5fff84f523c" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 36595 } } }
I20260806 01:02:21.374444 12357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ad519ddc3504a51a833d5fff84f523c [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:21.374739 12358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ad519ddc3504a51a833d5fff84f523c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ad519ddc3504a51a833d5fff84f523c. Latest consensus state: current_term: 1 leader_uuid: "3ad519ddc3504a51a833d5fff84f523c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ad519ddc3504a51a833d5fff84f523c" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 36595 } } }
I20260806 01:02:21.374876 12358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ad519ddc3504a51a833d5fff84f523c [sys.catalog]: This master's current role is: LEADER
W20260806 01:02:21.375746 12372 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ad519ddc3504a51a833d5fff84f523c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 01:02:21.375818 12372 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 01:02:21.375881 12365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:02:21.376068 12365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:02:21.378018 12355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.370430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42884 (local address 127.23.242.62:36595)
0806 01:02:21.370708 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:21.370719 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:21.370778 (+ 59us) server_negotiation.cc:407] Connection header received
0806 01:02:21.371425 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:21.371435 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:21.371524 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:21.371705 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:21.371997 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.373941 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.375382 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.376075 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.376250 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.377240 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:21.377245 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:21.377245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:21.377462 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:21.377508 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:21.377525 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:21.377597 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:21.377706 (+ 109us) server_negotiation.cc:299] Negotiation successful
0806 01:02:21.377807 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":99,"threads_started":1}
I20260806 01:02:21.378500 12365 catalog_manager.cc:1347] Generated new cluster ID: 7ab054bea56742cd9a91ea51fdddfb87
I20260806 01:02:21.378595 12365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:02:21.414865 12365 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:02:21.415474 12365 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:02:21.423115 12365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ad519ddc3504a51a833d5fff84f523c: Generated new TSK 0
I20260806 01:02:21.423352 12365 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 01:02:21.436522 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.1:0
--local_ip_for_outbound_sockets=127.23.242.1
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:36595
--builtin_ntp_servers=127.23.242.20:43695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 01:02:21.533743 12375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:21.533986 12375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:21.534013 12375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:21.535477 12375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:21.535528 12375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:21.535559 12375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.1
I20260806 01:02:21.537163 12375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:43695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.242.1
--webserver_port=0
--tserver_master_addrs=127.23.242.62:36595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.1
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:21.537472 12375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:21.537858 12375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:21.543676 12382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:21.543668 12381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:21.543630 12384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:21.543998 12375 server_base.cc:1034] running on GCE node
I20260806 01:02:21.544179 12375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:21.544405 12375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:21.545646 12375 hybrid_clock.cc:648] HybridClock initialized: now 1785978141545625 us; error 29 us; skew 500 ppm
I20260806 01:02:21.547158 12375 webserver.cc:460] Webserver started at http://127.23.242.1:38171/ using document root <none> and password file <none>
I20260806 01:02:21.547439 12375 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:21.547535 12375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:21.547719 12375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:21.548962 12375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-0/data/instance:
uuid: "92d8adc612a74f9e8e3507c800a0332a"
format_stamp: "Formatted at 2026-08-06 01:02:21 on dist-test-slave-dhph"
server_key: "8f6d6b86e3481e3b7c2e5c1f820d95fb"
server_key_iv: "86d8baf76e35b64ed00e5e323334e077"
server_key_version: "encryptionkey@0"
I20260806 01:02:21.549366 12375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance:
uuid: "92d8adc612a74f9e8e3507c800a0332a"
format_stamp: "Formatted at 2026-08-06 01:02:21 on dist-test-slave-dhph"
server_key: "8f6d6b86e3481e3b7c2e5c1f820d95fb"
server_key_iv: "86d8baf76e35b64ed00e5e323334e077"
server_key_version: "encryptionkey@0"
I20260806 01:02:21.551005 12375 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260806 01:02:21.551980 12390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:21.552213 12375 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:21.552328 12375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-0/wal
uuid: "92d8adc612a74f9e8e3507c800a0332a"
format_stamp: "Formatted at 2026-08-06 01:02:21 on dist-test-slave-dhph"
server_key: "8f6d6b86e3481e3b7c2e5c1f820d95fb"
server_key_iv: "86d8baf76e35b64ed00e5e323334e077"
server_key_version: "encryptionkey@0"
I20260806 01:02:21.552412 12375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:21.565387 12375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:21.565784 12375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:21.565956 12375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:21.566226 12375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:02:21.566635 12375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:02:21.566712 12375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:21.566767 12375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:02:21.566795 12375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:21.574450 12375 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.1:41845
I20260806 01:02:21.574611 12511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.1:41845 every 8 connection(s)
I20260806 01:02:21.575038 12375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-0/data/info.pb
I20260806 01:02:21.581521 12355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.576162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.1:53151 (local address 127.23.242.62:36595)
0806 01:02:21.576231 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:21.576233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:21.576457 (+ 224us) server_negotiation.cc:407] Connection header received
0806 01:02:21.576998 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:21.577000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:21.577017 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:21.577091 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:21.577878 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.578437 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.579713 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.579933 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.580007 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.581138 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:21.581139 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:21.581139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:21.581145 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:21.581161 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:21.581165 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:21.581226 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:21.581418 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 01:02:21.581447 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 01:02:21.581758 12514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.576070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:36595 (local address 127.23.242.1:53151)
0806 01:02:21.576297 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:21.576338 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:21.576835 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:21.577191 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:21.577195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:21.577203 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:21.577768 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.577773 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.578545 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.578546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.579552 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.579555 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.580076 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.580078 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.580215 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.580962 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 01:02:21.580963 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:21.580995 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 01:02:21.581024 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:21.581293 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:21.581300 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:21.581303 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 01:02:21.581377 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 01:02:21.581472 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":114,"threads_started":1}
I20260806 01:02:21.582293 12512 heartbeater.cc:347] Connected to a master server at 127.23.242.62:36595
I20260806 01:02:21.582401 12512 heartbeater.cc:464] Registering TS with master...
I20260806 01:02:21.582561 12512 heartbeater.cc:511] Master 127.23.242.62:36595 requested a full tablet report, sending...
I20260806 01:02:21.583164 12313 ts_manager.cc:194] Registered new tserver with Master: 92d8adc612a74f9e8e3507c800a0332a (127.23.242.1:41845)
I20260806 01:02:21.583590 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 12375
I20260806 01:02:21.583714 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-0/wal/instance
I20260806 01:02:21.583911 24520 external_mini_cluster.cc:1428] Setting key a54741acc962341156047635a827bfd1
I20260806 01:02:21.584085 12313 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.242.1:53151
I20260806 01:02:21.585019 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.2:0
--local_ip_for_outbound_sockets=127.23.242.2
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:36595
--builtin_ntp_servers=127.23.242.20:43695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 01:02:21.688267 12515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:21.688525 12515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:21.688565 12515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:21.689736 12515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:21.689805 12515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:21.689852 12515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.2
I20260806 01:02:21.691298 12515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:43695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.242.2
--webserver_port=0
--tserver_master_addrs=127.23.242.62:36595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.2
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:21.691568 12515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:21.691895 12515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:21.697906 12521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:21.697906 12522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:21.697908 12524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:21.698137 12515 server_base.cc:1034] running on GCE node
I20260806 01:02:21.698431 12515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:21.698767 12515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:21.699970 12515 hybrid_clock.cc:648] HybridClock initialized: now 1785978141699933 us; error 60 us; skew 500 ppm
I20260806 01:02:21.701604 12515 webserver.cc:460] Webserver started at http://127.23.242.2:33487/ using document root <none> and password file <none>
I20260806 01:02:21.701902 12515 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:21.701983 12515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:21.702172 12515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:21.703477 12515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-1/data/instance:
uuid: "5a2ab4ef14fd4250a54afde676f33968"
format_stamp: "Formatted at 2026-08-06 01:02:21 on dist-test-slave-dhph"
server_key: "4046e2b13b6ac99c980e408b9de7fabb"
server_key_iv: "31f89a8eac0b1dadf8cac0473a48c85e"
server_key_version: "encryptionkey@0"
I20260806 01:02:21.703884 12515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance:
uuid: "5a2ab4ef14fd4250a54afde676f33968"
format_stamp: "Formatted at 2026-08-06 01:02:21 on dist-test-slave-dhph"
server_key: "4046e2b13b6ac99c980e408b9de7fabb"
server_key_iv: "31f89a8eac0b1dadf8cac0473a48c85e"
server_key_version: "encryptionkey@0"
I20260806 01:02:21.705426 12515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 01:02:21.706549 12531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:21.706984 12515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:21.707120 12515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-1/wal
uuid: "5a2ab4ef14fd4250a54afde676f33968"
format_stamp: "Formatted at 2026-08-06 01:02:21 on dist-test-slave-dhph"
server_key: "4046e2b13b6ac99c980e408b9de7fabb"
server_key_iv: "31f89a8eac0b1dadf8cac0473a48c85e"
server_key_version: "encryptionkey@0"
I20260806 01:02:21.707209 12515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:21.731534 12515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:21.731951 12515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:21.732146 12515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:21.732442 12515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:02:21.733444 12515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:02:21.733518 12515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:21.733568 12515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:02:21.733623 12515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:21.740657 12515 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.2:44619
I20260806 01:02:21.740877 12650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.2:44619 every 8 connection(s)
I20260806 01:02:21.741190 12515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-1/data/info.pb
I20260806 01:02:21.742928 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 12515
I20260806 01:02:21.743062 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-1/wal/instance
I20260806 01:02:21.743273 24520 external_mini_cluster.cc:1428] Setting key 6a6cc89b1140e3b6b2246aa1b7cdd091
I20260806 01:02:21.744352 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.242.3:0
--local_ip_for_outbound_sockets=127.23.242.3
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:36595
--builtin_ntp_servers=127.23.242.20:43695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 01:02:21.747817 12355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.742074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.2:49803 (local address 127.23.242.62:36595)
0806 01:02:21.742150 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:21.742152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:21.742446 (+ 294us) server_negotiation.cc:407] Connection header received
0806 01:02:21.743304 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:21.743306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:21.743319 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:21.743385 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:21.744263 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.744956 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.746161 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.746391 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.746474 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.747489 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:21.747490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:21.747491 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:21.747496 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:21.747514 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:21.747518 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:21.747574 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:21.747705 (+ 131us) server_negotiation.cc:299] Negotiation successful
0806 01:02:21.747734 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 01:02:21.748064 12653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.742052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:36595 (local address 127.23.242.2:49803)
0806 01:02:21.742297 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:21.742348 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:21.743142 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:21.743464 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:21.743468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:21.743479 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:21.744161 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.744167 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.745074 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.745075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.746021 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.746024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.746485 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.746487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.746697 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.747314 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 01:02:21.747315 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:21.747345 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 01:02:21.747369 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:21.747577 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:21.747582 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:21.747586 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 01:02:21.747707 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 01:02:21.747819 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":95,"threads_started":1}
I20260806 01:02:21.748679 12651 heartbeater.cc:347] Connected to a master server at 127.23.242.62:36595
I20260806 01:02:21.748785 12651 heartbeater.cc:464] Registering TS with master...
I20260806 01:02:21.748997 12651 heartbeater.cc:511] Master 127.23.242.62:36595 requested a full tablet report, sending...
I20260806 01:02:21.749446 12313 ts_manager.cc:194] Registered new tserver with Master: 5a2ab4ef14fd4250a54afde676f33968 (127.23.242.2:44619)
I20260806 01:02:21.750017 12313 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.242.2:49803
W20260806 01:02:21.844398 12654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:21.844660 12654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:21.844688 12654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:21.846052 12654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:21.846098 12654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:21.846129 12654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.242.3
I20260806 01:02:21.847786 12654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:43695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.242.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.242.3
--webserver_port=0
--tserver_master_addrs=127.23.242.62:36595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.242.3
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:21.848022 12654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:21.848333 12654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:21.854559 12662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:21.854545 12660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:21.854550 12659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:21.855448 12654 server_base.cc:1034] running on GCE node
I20260806 01:02:21.855899 12654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:21.856514 12654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:21.857772 12654 hybrid_clock.cc:648] HybridClock initialized: now 1785978141857736 us; error 38 us; skew 500 ppm
I20260806 01:02:21.859915 12654 webserver.cc:460] Webserver started at http://127.23.242.3:34079/ using document root <none> and password file <none>
I20260806 01:02:21.860241 12654 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:21.860302 12654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:21.860411 12654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:21.861709 12654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-2/data/instance:
uuid: "2337b26f4e374292af5bca87d5eeb2bf"
format_stamp: "Formatted at 2026-08-06 01:02:21 on dist-test-slave-dhph"
server_key: "c322ed38314347301685b1a4f86358cd"
server_key_iv: "69a8bb6243efc03b9ba032a3af98b129"
server_key_version: "encryptionkey@0"
I20260806 01:02:21.862152 12654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance:
uuid: "2337b26f4e374292af5bca87d5eeb2bf"
format_stamp: "Formatted at 2026-08-06 01:02:21 on dist-test-slave-dhph"
server_key: "c322ed38314347301685b1a4f86358cd"
server_key_iv: "69a8bb6243efc03b9ba032a3af98b129"
server_key_version: "encryptionkey@0"
I20260806 01:02:21.863879 12654 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260806 01:02:21.865069 12668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:21.865456 12654 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:21.865577 12654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-2/wal
uuid: "2337b26f4e374292af5bca87d5eeb2bf"
format_stamp: "Formatted at 2026-08-06 01:02:21 on dist-test-slave-dhph"
server_key: "c322ed38314347301685b1a4f86358cd"
server_key_iv: "69a8bb6243efc03b9ba032a3af98b129"
server_key_version: "encryptionkey@0"
I20260806 01:02:21.865636 12654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:21.876844 12654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:21.877158 12654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:21.877282 12654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:21.877472 12654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 01:02:21.877827 12654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 01:02:21.877863 12654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:21.877888 12654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 01:02:21.877902 12654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:21.885964 12654 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.3:35643
I20260806 01:02:21.886080 12785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.3:35643 every 8 connection(s)
I20260806 01:02:21.886417 12654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-2/data/info.pb
I20260806 01:02:21.891968 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 12654
I20260806 01:02:21.892122 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-2/wal/instance
I20260806 01:02:21.892349 24520 external_mini_cluster.cc:1428] Setting key e908c7121b696d1a3caf9b8ed24972e7
I20260806 01:02:21.893888 12355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.887783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:58905 (local address 127.23.242.62:36595)
0806 01:02:21.887912 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:21.887914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:21.888459 (+ 545us) server_negotiation.cc:407] Connection header received
0806 01:02:21.888962 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:21.888963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:21.888974 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:21.889041 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:21.889755 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.890257 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.891758 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.891980 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.892050 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.893491 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:21.893493 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:21.893493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:21.893499 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:21.893516 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:21.893523 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:21.893657 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:21.893753 (+ 96us) server_negotiation.cc:299] Negotiation successful
0806 01:02:21.893801 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260806 01:02:21.894240 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.887782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:36595 (local address 127.23.242.3:58905)
0806 01:02:21.888289 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:21.888328 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:21.888859 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:21.889094 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:21.889097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:21.889104 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:21.889617 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.889622 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.890483 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.890484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.891591 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.891594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.892163 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.892165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.892369 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.893257 (+ 888us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 01:02:21.893258 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:21.893308 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 01:02:21.893358 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:21.893657 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:21.893666 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:21.893670 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 01:02:21.893744 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 01:02:21.893866 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":360,"threads_started":1}
I20260806 01:02:21.894964 12786 heartbeater.cc:347] Connected to a master server at 127.23.242.62:36595
I20260806 01:02:21.895069 12786 heartbeater.cc:464] Registering TS with master...
I20260806 01:02:21.895292 12786 heartbeater.cc:511] Master 127.23.242.62:36595 requested a full tablet report, sending...
I20260806 01:02:21.895915 12313 ts_manager.cc:194] Registered new tserver with Master: 2337b26f4e374292af5bca87d5eeb2bf (127.23.242.3:35643)
I20260806 01:02:21.896370 12313 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.242.3:58905
I20260806 01:02:21.905799 24520 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 01:02:21.912808 12355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.907999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42898 (local address 127.23.242.62:36595)
0806 01:02:21.908094 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:21.908095 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:21.908104 (+ 9us) server_negotiation.cc:407] Connection header received
0806 01:02:21.908128 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:21.908129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:21.908141 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:21.908240 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:21.908657 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.909501 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.911016 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.911244 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.911339 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.912501 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:21.912502 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:21.912502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:21.912515 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:21.912535 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:21.912540 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:21.912598 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:21.912663 (+ 65us) server_negotiation.cc:299] Negotiation successful
0806 01:02:21.912705 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 01:02:21.915886 12313 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 01:02:21.918007 12313 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 01:02:21.926178 12799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.921962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36038 (local address 127.23.242.1:41845)
0806 01:02:21.922569 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:21.922573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:21.922610 (+ 37us) server_negotiation.cc:407] Connection header received
0806 01:02:21.922648 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:21.922650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:21.922666 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:21.922741 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:21.922874 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.923909 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.925113 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.925729 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.925807 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.925901 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:21.926040 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 01:02:21.926085 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":96,"threads_started":1}
I20260806 01:02:21.926232 12794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.921160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:41845 (local address 127.0.0.1:36038)
0806 01:02:21.921351 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:21.921394 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:21.921461 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:21.922750 (+ 1289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:21.922751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:21.922752 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:21.922798 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.922799 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.924004 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.924005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.925021 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.925024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.925844 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.925845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.925908 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.925920 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 01:02:21.926018 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 01:02:21.926075 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":80,"thread_start_us":136,"threads_started":1}
I20260806 01:02:21.926826 12798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.921923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46698 (local address 127.23.242.2:44619)
0806 01:02:21.922154 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:21.922159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:21.922291 (+ 132us) server_negotiation.cc:407] Connection header received
0806 01:02:21.922338 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:21.922340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:21.922355 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:21.922435 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:21.922685 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.923482 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.925526 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.926278 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.926356 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.926445 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:21.926691 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 01:02:21.926738 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":102,"threads_started":1}
I20260806 01:02:21.927100 12795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.921467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:44619 (local address 127.0.0.1:46698)
0806 01:02:21.922148 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:21.922175 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:21.922230 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:21.922468 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:21.922469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:21.922471 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:21.922523 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.922524 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.923586 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.923587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.925425 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.925427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.926481 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.926482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.926547 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.926550 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 01:02:21.926964 (+ 414us) client_negotiation.cc:240] Negotiation successful
0806 01:02:21.927014 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":52,"thread_start_us":66,"threads_started":1}
I20260806 01:02:21.927203 12796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.921233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.3:35643 (local address 127.0.0.1:44354)
0806 01:02:21.921652 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:21.921673 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:21.921739 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:21.922132 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:21.922135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:21.922138 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:21.922242 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.922245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.923846 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.923848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.925424 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.925427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.926973 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.926974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.927043 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.927046 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 01:02:21.927076 (+ 30us) client_negotiation.cc:240] Negotiation successful
0806 01:02:21.927114 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":186,"thread_start_us":75,"threads_started":1}
I20260806 01:02:21.927579 12797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.921748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44354 (local address 127.23.242.3:35643)
0806 01:02:21.921926 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:21.921931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:21.921942 (+ 11us) server_negotiation.cc:407] Connection header received
0806 01:02:21.921984 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:21.921986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:21.922003 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:21.922326 (+ 323us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:21.922385 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.923698 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.925953 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.926869 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.927306 (+ 437us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.927424 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:21.927453 (+ 29us) server_negotiation.cc:299] Negotiation successful
0806 01:02:21.927498 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":108,"threads_started":1}
I20260806 01:02:21.927450 12440 tablet_service.cc:1467] Processing CreateTablet for tablet c14a57e86f694f5687cf14c14a578704 (DEFAULT_TABLE table=test-table [id=fdfd2dc967534ebeb1fa02cd5868e2e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:02:21.927911 12440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c14a57e86f694f5687cf14c14a578704. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:21.927839 12581 tablet_service.cc:1467] Processing CreateTablet for tablet c14a57e86f694f5687cf14c14a578704 (DEFAULT_TABLE table=test-table [id=fdfd2dc967534ebeb1fa02cd5868e2e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:02:21.928161 12581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c14a57e86f694f5687cf14c14a578704. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:21.928211 12717 tablet_service.cc:1467] Processing CreateTablet for tablet c14a57e86f694f5687cf14c14a578704 (DEFAULT_TABLE table=test-table [id=fdfd2dc967534ebeb1fa02cd5868e2e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 01:02:21.928522 12717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c14a57e86f694f5687cf14c14a578704. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:21.931615 12800 tablet_bootstrap.cc:492] T c14a57e86f694f5687cf14c14a578704 P 5a2ab4ef14fd4250a54afde676f33968: Bootstrap starting.
I20260806 01:02:21.932827 12800 tablet_bootstrap.cc:654] T c14a57e86f694f5687cf14c14a578704 P 5a2ab4ef14fd4250a54afde676f33968: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:21.933272 12800 log.cc:824] T c14a57e86f694f5687cf14c14a578704 P 5a2ab4ef14fd4250a54afde676f33968: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:21.933408 12802 tablet_bootstrap.cc:492] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf: Bootstrap starting.
I20260806 01:02:21.933679 12801 tablet_bootstrap.cc:492] T c14a57e86f694f5687cf14c14a578704 P 92d8adc612a74f9e8e3507c800a0332a: Bootstrap starting.
I20260806 01:02:21.934214 12800 tablet_bootstrap.cc:492] T c14a57e86f694f5687cf14c14a578704 P 5a2ab4ef14fd4250a54afde676f33968: No bootstrap required, opened a new log
I20260806 01:02:21.934296 12800 ts_tablet_manager.cc:1397] T c14a57e86f694f5687cf14c14a578704 P 5a2ab4ef14fd4250a54afde676f33968: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 01:02:21.934351 12802 tablet_bootstrap.cc:654] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:21.934430 12801 tablet_bootstrap.cc:654] T c14a57e86f694f5687cf14c14a578704 P 92d8adc612a74f9e8e3507c800a0332a: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:21.934716 12802 log.cc:824] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:21.934974 12801 log.cc:824] T c14a57e86f694f5687cf14c14a578704 P 92d8adc612a74f9e8e3507c800a0332a: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:21.935580 12802 tablet_bootstrap.cc:492] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf: No bootstrap required, opened a new log
I20260806 01:02:21.935699 12802 ts_tablet_manager.cc:1397] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 01:02:21.936004 12801 tablet_bootstrap.cc:492] T c14a57e86f694f5687cf14c14a578704 P 92d8adc612a74f9e8e3507c800a0332a: No bootstrap required, opened a new log
I20260806 01:02:21.936074 12801 ts_tablet_manager.cc:1397] T c14a57e86f694f5687cf14c14a578704 P 92d8adc612a74f9e8e3507c800a0332a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260806 01:02:21.937291 12800 raft_consensus.cc:348] T c14a57e86f694f5687cf14c14a578704 P 5a2ab4ef14fd4250a54afde676f33968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d8adc612a74f9e8e3507c800a0332a" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41845 } } peers { permanent_uuid: "2337b26f4e374292af5bca87d5eeb2bf" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 35643 } } peers { permanent_uuid: "5a2ab4ef14fd4250a54afde676f33968" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 44619 } }
I20260806 01:02:21.937490 12800 raft_consensus.cc:374] T c14a57e86f694f5687cf14c14a578704 P 5a2ab4ef14fd4250a54afde676f33968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:21.937539 12800 raft_consensus.cc:729] T c14a57e86f694f5687cf14c14a578704 P 5a2ab4ef14fd4250a54afde676f33968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a2ab4ef14fd4250a54afde676f33968, State: Initialized, Role: FOLLOWER
I20260806 01:02:21.937700 12800 consensus_queue.cc:260] T c14a57e86f694f5687cf14c14a578704 P 5a2ab4ef14fd4250a54afde676f33968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d8adc612a74f9e8e3507c800a0332a" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41845 } } peers { permanent_uuid: "2337b26f4e374292af5bca87d5eeb2bf" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 35643 } } peers { permanent_uuid: "5a2ab4ef14fd4250a54afde676f33968" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 44619 } }
I20260806 01:02:21.937996 12800 ts_tablet_manager.cc:1428] T c14a57e86f694f5687cf14c14a578704 P 5a2ab4ef14fd4250a54afde676f33968: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260806 01:02:21.938122 12651 heartbeater.cc:503] Master 127.23.242.62:36595 was elected leader, sending a full tablet report...
I20260806 01:02:21.938764 12801 raft_consensus.cc:348] T c14a57e86f694f5687cf14c14a578704 P 92d8adc612a74f9e8e3507c800a0332a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d8adc612a74f9e8e3507c800a0332a" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41845 } } peers { permanent_uuid: "2337b26f4e374292af5bca87d5eeb2bf" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 35643 } } peers { permanent_uuid: "5a2ab4ef14fd4250a54afde676f33968" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 44619 } }
I20260806 01:02:21.938951 12801 raft_consensus.cc:374] T c14a57e86f694f5687cf14c14a578704 P 92d8adc612a74f9e8e3507c800a0332a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:21.939002 12801 raft_consensus.cc:729] T c14a57e86f694f5687cf14c14a578704 P 92d8adc612a74f9e8e3507c800a0332a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92d8adc612a74f9e8e3507c800a0332a, State: Initialized, Role: FOLLOWER
I20260806 01:02:21.939142 12801 consensus_queue.cc:260] T c14a57e86f694f5687cf14c14a578704 P 92d8adc612a74f9e8e3507c800a0332a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d8adc612a74f9e8e3507c800a0332a" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41845 } } peers { permanent_uuid: "2337b26f4e374292af5bca87d5eeb2bf" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 35643 } } peers { permanent_uuid: "5a2ab4ef14fd4250a54afde676f33968" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 44619 } }
I20260806 01:02:21.939571 12801 ts_tablet_manager.cc:1428] T c14a57e86f694f5687cf14c14a578704 P 92d8adc612a74f9e8e3507c800a0332a: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260806 01:02:21.939827 12512 heartbeater.cc:503] Master 127.23.242.62:36595 was elected leader, sending a full tablet report...
I20260806 01:02:21.940393 12802 raft_consensus.cc:348] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d8adc612a74f9e8e3507c800a0332a" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41845 } } peers { permanent_uuid: "2337b26f4e374292af5bca87d5eeb2bf" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 35643 } } peers { permanent_uuid: "5a2ab4ef14fd4250a54afde676f33968" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 44619 } }
I20260806 01:02:21.940732 12802 raft_consensus.cc:374] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:21.940876 12802 raft_consensus.cc:729] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2337b26f4e374292af5bca87d5eeb2bf, State: Initialized, Role: FOLLOWER
I20260806 01:02:21.941150 12802 consensus_queue.cc:260] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d8adc612a74f9e8e3507c800a0332a" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41845 } } peers { permanent_uuid: "2337b26f4e374292af5bca87d5eeb2bf" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 35643 } } peers { permanent_uuid: "5a2ab4ef14fd4250a54afde676f33968" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 44619 } }
I20260806 01:02:21.941478 12802 ts_tablet_manager.cc:1428] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 01:02:21.941620 12786 heartbeater.cc:503] Master 127.23.242.62:36595 was elected leader, sending a full tablet report...
W20260806 01:02:21.992643 12652 tablet.cc:2369] T c14a57e86f694f5687cf14c14a578704 P 5a2ab4ef14fd4250a54afde676f33968: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 01:02:21.994864 12808 raft_consensus.cc:482] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 01:02:21.994982 12808 raft_consensus.cc:504] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d8adc612a74f9e8e3507c800a0332a" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41845 } } peers { permanent_uuid: "2337b26f4e374292af5bca87d5eeb2bf" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 35643 } } peers { permanent_uuid: "5a2ab4ef14fd4250a54afde676f33968" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 44619 } }
I20260806 01:02:21.995494 12808 leader_election.cc:288] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92d8adc612a74f9e8e3507c800a0332a (127.23.242.1:41845), 5a2ab4ef14fd4250a54afde676f33968 (127.23.242.2:44619)
I20260806 01:02:21.998871 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.995752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:41845 (local address 127.23.242.3:33519)
0806 01:02:21.995923 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:21.995938 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:21.996006 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:21.996173 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:21.996174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:21.996176 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:21.996255 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.996257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.997214 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.997215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.997961 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.997963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.998566 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.998567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.998667 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.998670 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 01:02:21.998742 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 01:02:21.998790 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260806 01:02:21.998871 12799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.995935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:33519 (local address 127.23.242.1:41845)
0806 01:02:21.995993 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:21.995997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:21.996011 (+ 14us) server_negotiation.cc:407] Connection header received
0806 01:02:21.996072 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:21.996073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:21.996086 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:21.996151 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:21.996363 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.997137 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.998063 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.998473 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.998565 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.998575 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:21.998761 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 01:02:21.998805 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260806 01:02:21.999302 12809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.995767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.2:44619 (local address 127.23.242.3:44669)
0806 01:02:21.995985 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:21.995997 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:21.996068 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:21.996361 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:21.996362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:21.996365 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 01:02:21.996440 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.996442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.997456 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.997457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.998339 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:21.998342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.999049 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.999050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:21.999118 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.999123 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 01:02:21.999180 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 01:02:21.999218 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":3,"thread_start_us":133,"threads_started":1}
I20260806 01:02:21.999302 12798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:21.995956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.242.3:44669 (local address 127.23.242.2:44619)
0806 01:02:21.996001 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:21.996003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:21.996055 (+ 52us) server_negotiation.cc:407] Connection header received
0806 01:02:21.996218 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:21.996220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:21.996233 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:21.996300 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 01:02:21.996601 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.997363 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.998446 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:21.998869 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:21.998966 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:21.998977 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:21.999186 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 01:02:21.999235 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 01:02:21.999280 12463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c14a57e86f694f5687cf14c14a578704" candidate_uuid: "2337b26f4e374292af5bca87d5eeb2bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92d8adc612a74f9e8e3507c800a0332a" is_pre_election: true
I20260806 01:02:21.999491 12463 raft_consensus.cc:2454] T c14a57e86f694f5687cf14c14a578704 P 92d8adc612a74f9e8e3507c800a0332a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2337b26f4e374292af5bca87d5eeb2bf in term 0.
I20260806 01:02:21.999635 12602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c14a57e86f694f5687cf14c14a578704" candidate_uuid: "2337b26f4e374292af5bca87d5eeb2bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2ab4ef14fd4250a54afde676f33968" is_pre_election: true
I20260806 01:02:21.999781 12602 raft_consensus.cc:2454] T c14a57e86f694f5687cf14c14a578704 P 5a2ab4ef14fd4250a54afde676f33968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2337b26f4e374292af5bca87d5eeb2bf in term 0.
I20260806 01:02:21.999790 12671 leader_election.cc:302] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2337b26f4e374292af5bca87d5eeb2bf, 92d8adc612a74f9e8e3507c800a0332a; no voters:
I20260806 01:02:21.999970 12808 raft_consensus.cc:2781] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 01:02:22.000030 12808 raft_consensus.cc:482] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 01:02:22.000065 12808 raft_consensus.cc:3037] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:22.000818 12808 raft_consensus.cc:504] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d8adc612a74f9e8e3507c800a0332a" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41845 } } peers { permanent_uuid: "2337b26f4e374292af5bca87d5eeb2bf" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 35643 } } peers { permanent_uuid: "5a2ab4ef14fd4250a54afde676f33968" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 44619 } }
I20260806 01:02:22.000965 12808 leader_election.cc:288] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf [CANDIDATE]: Term 1 election: Requested vote from peers 92d8adc612a74f9e8e3507c800a0332a (127.23.242.1:41845), 5a2ab4ef14fd4250a54afde676f33968 (127.23.242.2:44619)
I20260806 01:02:22.001168 12602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c14a57e86f694f5687cf14c14a578704" candidate_uuid: "2337b26f4e374292af5bca87d5eeb2bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a2ab4ef14fd4250a54afde676f33968"
I20260806 01:02:22.001260 12602 raft_consensus.cc:3037] T c14a57e86f694f5687cf14c14a578704 P 5a2ab4ef14fd4250a54afde676f33968 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:22.001436 12463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c14a57e86f694f5687cf14c14a578704" candidate_uuid: "2337b26f4e374292af5bca87d5eeb2bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92d8adc612a74f9e8e3507c800a0332a"
I20260806 01:02:22.001606 12463 raft_consensus.cc:3037] T c14a57e86f694f5687cf14c14a578704 P 92d8adc612a74f9e8e3507c800a0332a [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:22.002118 12602 raft_consensus.cc:2454] T c14a57e86f694f5687cf14c14a578704 P 5a2ab4ef14fd4250a54afde676f33968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2337b26f4e374292af5bca87d5eeb2bf in term 1.
I20260806 01:02:22.002314 12672 leader_election.cc:302] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2337b26f4e374292af5bca87d5eeb2bf, 5a2ab4ef14fd4250a54afde676f33968; no voters:
I20260806 01:02:22.002421 12808 raft_consensus.cc:2781] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:22.002635 12808 raft_consensus.cc:686] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf [term 1 LEADER]: Becoming Leader. State: Replica: 2337b26f4e374292af5bca87d5eeb2bf, State: Running, Role: LEADER
I20260806 01:02:22.002746 12808 consensus_queue.cc:237] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d8adc612a74f9e8e3507c800a0332a" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41845 } } peers { permanent_uuid: "2337b26f4e374292af5bca87d5eeb2bf" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 35643 } } peers { permanent_uuid: "5a2ab4ef14fd4250a54afde676f33968" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 44619 } }
I20260806 01:02:22.003413 12463 raft_consensus.cc:2454] T c14a57e86f694f5687cf14c14a578704 P 92d8adc612a74f9e8e3507c800a0332a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2337b26f4e374292af5bca87d5eeb2bf in term 1.
I20260806 01:02:22.004014 12313 catalog_manager.cc:5495] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 2337b26f4e374292af5bca87d5eeb2bf (127.23.242.3). New cstate: current_term: 1 leader_uuid: "2337b26f4e374292af5bca87d5eeb2bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d8adc612a74f9e8e3507c800a0332a" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2337b26f4e374292af5bca87d5eeb2bf" member_type: VOTER last_known_addr { host: "127.23.242.3" port: 35643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a2ab4ef14fd4250a54afde676f33968" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 44619 } health_report { overall_health: UNKNOWN } } }
I20260806 01:02:22.022959 12797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:22.019683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44362 (local address 127.23.242.3:35643)
0806 01:02:22.019762 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:22.019764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:22.019774 (+ 10us) server_negotiation.cc:407] Connection header received
0806 01:02:22.019827 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:22.019828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:22.019856 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:22.019937 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 01:02:22.020243 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:22.020726 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:22.021610 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:22.021878 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:22.021992 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:22.022312 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 01:02:22.022435 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 01:02:22.022515 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:22.022836 (+ 321us) server_negotiation.cc:299] Negotiation successful
0806 01:02:22.022871 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 01:02:22.025589 12463 raft_consensus.cc:1261] T c14a57e86f694f5687cf14c14a578704 P 92d8adc612a74f9e8e3507c800a0332a [term 1 FOLLOWER]: Refusing update from remote peer 2337b26f4e374292af5bca87d5eeb2bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:02:22.025589 12602 raft_consensus.cc:1261] T c14a57e86f694f5687cf14c14a578704 P 5a2ab4ef14fd4250a54afde676f33968 [term 1 FOLLOWER]: Refusing update from remote peer 2337b26f4e374292af5bca87d5eeb2bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:02:22.026013 12810 consensus_queue.cc:1035] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "92d8adc612a74f9e8e3507c800a0332a" member_type: VOTER last_known_addr { host: "127.23.242.1" port: 41845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:02:22.026551 12808 consensus_queue.cc:1035] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a2ab4ef14fd4250a54afde676f33968" member_type: VOTER last_known_addr { host: "127.23.242.2" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:02:22.030637 12815 mvcc.cc:205] Tried to move back new op lower bound from 7315366469733203968 to 7315366469643993088. Current Snapshot: MvccSnapshot[applied={T|T < 7315366469733203968}]
I20260806 01:02:22.031409 12814 mvcc.cc:205] Tried to move back new op lower bound from 7315366469733203968 to 7315366469643993088. Current Snapshot: MvccSnapshot[applied={T|T < 7315366469733203968}]
I20260806 01:02:22.033526 12312 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 01:02:22.033645 12312 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 01:02:22.034725 12312 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ad519ddc3504a51a833d5fff84f523c: Sending DeleteTablet for 3 replicas of tablet c14a57e86f694f5687cf14c14a578704
I20260806 01:02:22.035210 12717 tablet_service.cc:1514] Processing DeleteTablet for tablet c14a57e86f694f5687cf14c14a578704 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:02:22 UTC) from {username='slave'} at 127.0.0.1:44354
I20260806 01:02:22.035233 12440 tablet_service.cc:1514] Processing DeleteTablet for tablet c14a57e86f694f5687cf14c14a578704 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:02:22 UTC) from {username='slave'} at 127.0.0.1:36038
I20260806 01:02:22.035589 12581 tablet_service.cc:1514] Processing DeleteTablet for tablet c14a57e86f694f5687cf14c14a578704 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 01:02:22 UTC) from {username='slave'} at 127.0.0.1:46698
I20260806 01:02:22.035858 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 12375
I20260806 01:02:22.035892 12824 tablet_replica.cc:330] T c14a57e86f694f5687cf14c14a578704 P 5a2ab4ef14fd4250a54afde676f33968: stopping tablet replica
I20260806 01:02:22.036015 12824 raft_consensus.cc:2229] T c14a57e86f694f5687cf14c14a578704 P 5a2ab4ef14fd4250a54afde676f33968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 01:02:22.036137 12824 raft_consensus.cc:2258] T c14a57e86f694f5687cf14c14a578704 P 5a2ab4ef14fd4250a54afde676f33968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 01:02:22.037868 12824 ts_tablet_manager.cc:1905] T c14a57e86f694f5687cf14c14a578704 P 5a2ab4ef14fd4250a54afde676f33968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 01:02:22.040519 12824 ts_tablet_manager.cc:1918] T c14a57e86f694f5687cf14c14a578704 P 5a2ab4ef14fd4250a54afde676f33968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 01:02:22.040649 12824 log.cc:1196] T c14a57e86f694f5687cf14c14a578704 P 5a2ab4ef14fd4250a54afde676f33968: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-1/wal/wals/c14a57e86f694f5687cf14c14a578704
I20260806 01:02:22.041006 12824 ts_tablet_manager.cc:1939] T c14a57e86f694f5687cf14c14a578704 P 5a2ab4ef14fd4250a54afde676f33968: Deleting consensus metadata
I20260806 01:02:22.041632 12299 catalog_manager.cc:4841] TS 5a2ab4ef14fd4250a54afde676f33968 (127.23.242.2:44619): tablet c14a57e86f694f5687cf14c14a578704 (table test-table [id=fdfd2dc967534ebeb1fa02cd5868e2e9]) successfully deleted
I20260806 01:02:22.042726 12823 tablet_replica.cc:330] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf: stopping tablet replica
I20260806 01:02:22.042917 12823 raft_consensus.cc:2229] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf [term 1 LEADER]: Raft consensus shutting down.
I20260806 01:02:22.043090 12823 raft_consensus.cc:2258] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 01:02:22.043666 12823 ts_tablet_manager.cc:1905] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 01:02:22.046303 12823 ts_tablet_manager.cc:1918] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 01:02:22.046404 12823 log.cc:1196] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf: Deleting WAL directory at /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785977869987768-24520-0/minicluster-data/ts-2/wal/wals/c14a57e86f694f5687cf14c14a578704
I20260806 01:02:22.046751 12823 ts_tablet_manager.cc:1939] T c14a57e86f694f5687cf14c14a578704 P 2337b26f4e374292af5bca87d5eeb2bf: Deleting consensus metadata
I20260806 01:02:22.047499 12299 catalog_manager.cc:4841] TS 2337b26f4e374292af5bca87d5eeb2bf (127.23.242.3:35643): tablet c14a57e86f694f5687cf14c14a578704 (table test-table [id=fdfd2dc967534ebeb1fa02cd5868e2e9]) successfully deleted
W20260806 01:02:22.048538 12298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.242.1:41845 (error 108)
I20260806 01:02:22.048945 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 12515
I20260806 01:02:22.049095 12795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:22.048933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.1:41845 (local address 127.0.0.1:36054)
0806 01:02:22.049002 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:22.049055 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.1:41845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 01:02:22.050624 12298 catalog_manager.cc:4584] TS 92d8adc612a74f9e8e3507c800a0332a (127.23.242.1:41845): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c14a57e86f694f5687cf14c14a578704: Network error: Client connection negotiation failed: client connection to 127.23.242.1:41845: connect: Connection refused (error 111)
I20260806 01:02:22.057718 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 12654
I20260806 01:02:22.065606 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 12279
2026-08-06T01:02:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (887 ms)
[----------] 6 tests from EncryptionPolicyTest (10305 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 01:02:22 dist-test-slave-dhph krb5kdc[12828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 01:02:22 dist-test-slave-dhph krb5kdc[12828](info): set up 2 sockets
Aug 06 01:02:22 dist-test-slave-dhph krb5kdc[12828](info): commencing operation
krb5kdc: starting...
W20260806 01:02:23.371826 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.282s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 01:02:23 dist-test-slave-dhph krb5kdc[12828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T01:02:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:02:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 01:02:23.529057 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40473
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:23.635078 12841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:23.635612 12841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:23.635645 12841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:23.637215 12841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:23.637283 12841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:23.637300 12841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:23.637315 12841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:23.637331 12841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:23.640177 12841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40473
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:23.640930 12841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:23.641376 12841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:23.649091 12847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:23.649066 12846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:23.649040 12849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:23.649437 12841 server_base.cc:1034] running on GCE node
I20260806 01:02:23.649729 12841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:23.650103 12841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:23.651312 12841 hybrid_clock.cc:648] HybridClock initialized: now 1785978143651272 us; error 53 us; skew 500 ppm
Aug 06 01:02:23 dist-test-slave-dhph krb5kdc[12828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:23.654305 12841 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 01:02:23.654901 12841 webserver.cc:460] Webserver started at http://127.0.0.1:38599/ using document root <none> and password file <none>
I20260806 01:02:23.655195 12841 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:23.655249 12841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:23.655401 12841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:23.656625 12841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "18b990ae8d2a4541a6c8d1030a328c27"
format_stamp: "Formatted at 2026-08-06 01:02:23 on dist-test-slave-dhph"
server_key: "d7d8148e37341878fa43c0f94793357a"
server_key_iv: "af2b58fe9ccb09c82201e6fef4d99caf"
server_key_version: "encryptionkey@0"
I20260806 01:02:23.657039 12841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "18b990ae8d2a4541a6c8d1030a328c27"
format_stamp: "Formatted at 2026-08-06 01:02:23 on dist-test-slave-dhph"
server_key: "d7d8148e37341878fa43c0f94793357a"
server_key_iv: "af2b58fe9ccb09c82201e6fef4d99caf"
server_key_version: "encryptionkey@0"
I20260806 01:02:23.659978 12841 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 01:02:23.661423 12856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:23.662021 12841 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260806 01:02:23.662142 12841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "18b990ae8d2a4541a6c8d1030a328c27"
format_stamp: "Formatted at 2026-08-06 01:02:23 on dist-test-slave-dhph"
server_key: "d7d8148e37341878fa43c0f94793357a"
server_key_iv: "af2b58fe9ccb09c82201e6fef4d99caf"
server_key_version: "encryptionkey@0"
I20260806 01:02:23.662312 12841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:23.681459 12841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:23.684754 12841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:23.684930 12841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:23.691807 12841 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40473
I20260806 01:02:23.691848 12911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40473 every 8 connection(s)
I20260806 01:02:23.692296 12841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:02:23.693300 12912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:23.696007 12912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18b990ae8d2a4541a6c8d1030a328c27: Bootstrap starting.
I20260806 01:02:23.696907 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 12841
I20260806 01:02:23.697043 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:02:23.697033 12912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18b990ae8d2a4541a6c8d1030a328c27: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:23.697319 24520 external_mini_cluster.cc:1428] Setting key fdf23ea41d1e3252d069ead36db91f50
I20260806 01:02:23.697412 12912 log.cc:824] T 00000000000000000000000000000000 P 18b990ae8d2a4541a6c8d1030a328c27: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:23.698439 12912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18b990ae8d2a4541a6c8d1030a328c27: No bootstrap required, opened a new log
I20260806 01:02:23.702373 12912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18b990ae8d2a4541a6c8d1030a328c27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b990ae8d2a4541a6c8d1030a328c27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40473 } }
I20260806 01:02:23.702577 12912 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18b990ae8d2a4541a6c8d1030a328c27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:23.702668 12912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18b990ae8d2a4541a6c8d1030a328c27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18b990ae8d2a4541a6c8d1030a328c27, State: Initialized, Role: FOLLOWER
I20260806 01:02:23.702903 12912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18b990ae8d2a4541a6c8d1030a328c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b990ae8d2a4541a6c8d1030a328c27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40473 } }
I20260806 01:02:23.703020 12912 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18b990ae8d2a4541a6c8d1030a328c27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:02:23.703069 12912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18b990ae8d2a4541a6c8d1030a328c27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:02:23.703125 12912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18b990ae8d2a4541a6c8d1030a328c27 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:23.704831 12912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18b990ae8d2a4541a6c8d1030a328c27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b990ae8d2a4541a6c8d1030a328c27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40473 } }
I20260806 01:02:23.705017 12912 leader_election.cc:302] T 00000000000000000000000000000000 P 18b990ae8d2a4541a6c8d1030a328c27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18b990ae8d2a4541a6c8d1030a328c27; no voters:
I20260806 01:02:23.705417 12912 leader_election.cc:288] T 00000000000000000000000000000000 P 18b990ae8d2a4541a6c8d1030a328c27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:02:23.705781 12912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18b990ae8d2a4541a6c8d1030a328c27 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:23.706032 12917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18b990ae8d2a4541a6c8d1030a328c27 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:23.706418 12917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18b990ae8d2a4541a6c8d1030a328c27 [term 1 LEADER]: Becoming Leader. State: Replica: 18b990ae8d2a4541a6c8d1030a328c27, State: Running, Role: LEADER
I20260806 01:02:23.706527 12917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18b990ae8d2a4541a6c8d1030a328c27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b990ae8d2a4541a6c8d1030a328c27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40473 } }
I20260806 01:02:23.706965 12918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18b990ae8d2a4541a6c8d1030a328c27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18b990ae8d2a4541a6c8d1030a328c27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b990ae8d2a4541a6c8d1030a328c27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40473 } } }
I20260806 01:02:23.707060 12918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18b990ae8d2a4541a6c8d1030a328c27 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:23.707271 12919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18b990ae8d2a4541a6c8d1030a328c27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18b990ae8d2a4541a6c8d1030a328c27. Latest consensus state: current_term: 1 leader_uuid: "18b990ae8d2a4541a6c8d1030a328c27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b990ae8d2a4541a6c8d1030a328c27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40473 } } }
I20260806 01:02:23.707347 12919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18b990ae8d2a4541a6c8d1030a328c27 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:23.707495 12926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:02:23.708163 12926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:02:23.710147 12926 catalog_manager.cc:1347] Generated new cluster ID: 2fcd6568af344363ad0313f2047d4a47
I20260806 01:02:23.710211 12926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 01:02:23 dist-test-slave-dhph krb5kdc[12828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 01:02:23.716195 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:23.697997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51125 (local address 127.0.0.1:40473)
0806 01:02:23.698225 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:23.698233 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:23.698261 (+ 28us) server_negotiation.cc:407] Connection header received
0806 01:02:23.700074 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:23.700085 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:23.700153 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:23.701071 (+ 918us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:23.701790 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:23.704808 (+ 3018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:23.706375 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:23.707292 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:23.707398 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:23.711376 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:23.711381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:23.711381 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:23.711396 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:23.713806 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:23.714370 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:23.714375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:23.714375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:23.714509 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:23.714892 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:23.714893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:23.714894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:23.715280 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:23.715475 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:23.715616 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 01:02:23.715754 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":95,"threads_started":1}
I20260806 01:02:23.717046 12926 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:02:23.717586 24520 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 01:02:23.717823 12926 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:02:23.720522 12926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18b990ae8d2a4541a6c8d1030a328c27: Generated new TSK 0
I20260806 01:02:23.720731 12926 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 01:02:23.730760 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:23.720332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60911 (local address 127.0.0.1:40473)
0806 01:02:23.720560 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:23.720562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:23.720570 (+ 8us) server_negotiation.cc:407] Connection header received
0806 01:02:23.720593 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:23.720593 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:23.720604 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:23.720651 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:23.721714 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:23.722839 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:23.723768 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:23.724006 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:23.724097 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:23.725610 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:23.725611 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:23.725611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:23.725624 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:23.729110 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:23.729715 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:23.729716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:23.729717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:23.729789 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:23.730060 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:23.730061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:23.730061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:23.730202 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:23.730339 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:23.730498 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 01:02:23.730609 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206}
I20260806 01:02:23.734349 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 12841
W20260806 01:02:23.742130 12826 connection.cc:535] client connection to 127.0.0.1:40473 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40473): Connection reset by peer (error 104)
2026-08-06T01:02:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1666 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 01:02:23 dist-test-slave-dhph krb5kdc[12945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 01:02:23 dist-test-slave-dhph krb5kdc[12945](info): set up 2 sockets
Aug 06 01:02:23 dist-test-slave-dhph krb5kdc[12945](info): commencing operation
krb5kdc: starting...
W20260806 01:02:24.987875 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.231s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 01:02:25 dist-test-slave-dhph krb5kdc[12945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T01:02:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:02:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 01:02:25.157867 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33481
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:25.267318 12958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:25.267637 12958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:25.267689 12958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:25.269438 12958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:25.269583 12958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:25.269979 12958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:25.270037 12958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:25.270224 12958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:25.272343 12958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33481
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:25.272693 12958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:25.273027 12958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:25.279526 12963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:25.280290 12966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:25.280311 12964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:25.280795 12958 server_base.cc:1034] running on GCE node
I20260806 01:02:25.281093 12958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:25.281489 12958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:25.282810 12958 hybrid_clock.cc:648] HybridClock initialized: now 1785978145282788 us; error 58 us; skew 500 ppm
Aug 06 01:02:25 dist-test-slave-dhph krb5kdc[12945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:25.285859 12958 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 01:02:25.286463 12958 webserver.cc:460] Webserver started at http://127.0.0.1:34481/ using document root <none> and password file <none>
I20260806 01:02:25.286753 12958 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:25.286804 12958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:25.286950 12958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:25.288151 12958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "0a13aec6cad04092a51e094fc279f8f1"
format_stamp: "Formatted at 2026-08-06 01:02:25 on dist-test-slave-dhph"
server_key: "465a1e0f73d1feafe0a839d7a8b68c02"
server_key_iv: "3f9525f3b56e19e19ea49d16f2e4e757"
server_key_version: "encryptionkey@0"
I20260806 01:02:25.288561 12958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "0a13aec6cad04092a51e094fc279f8f1"
format_stamp: "Formatted at 2026-08-06 01:02:25 on dist-test-slave-dhph"
server_key: "465a1e0f73d1feafe0a839d7a8b68c02"
server_key_iv: "3f9525f3b56e19e19ea49d16f2e4e757"
server_key_version: "encryptionkey@0"
I20260806 01:02:25.290211 12958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 01:02:25.291404 12973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:25.291952 12958 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:25.292071 12958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "0a13aec6cad04092a51e094fc279f8f1"
format_stamp: "Formatted at 2026-08-06 01:02:25 on dist-test-slave-dhph"
server_key: "465a1e0f73d1feafe0a839d7a8b68c02"
server_key_iv: "3f9525f3b56e19e19ea49d16f2e4e757"
server_key_version: "encryptionkey@0"
I20260806 01:02:25.292271 12958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:25.302152 12958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:25.305294 12958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:25.305449 12958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:25.311961 12958 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33481
I20260806 01:02:25.312436 12958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:02:25.312425 13027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33481 every 8 connection(s)
I20260806 01:02:25.313808 13028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:25.316326 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 12958
I20260806 01:02:25.316468 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:02:25.316685 24520 external_mini_cluster.cc:1428] Setting key 6c70342559fbd485ca8213fd829ca628
I20260806 01:02:25.316929 13028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a13aec6cad04092a51e094fc279f8f1: Bootstrap starting.
I20260806 01:02:25.317759 13028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a13aec6cad04092a51e094fc279f8f1: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:25.318111 13028 log.cc:824] T 00000000000000000000000000000000 P 0a13aec6cad04092a51e094fc279f8f1: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:25.319830 13028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a13aec6cad04092a51e094fc279f8f1: No bootstrap required, opened a new log
I20260806 01:02:25.323604 13028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a13aec6cad04092a51e094fc279f8f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a13aec6cad04092a51e094fc279f8f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } }
I20260806 01:02:25.323923 13028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a13aec6cad04092a51e094fc279f8f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:25.323975 13028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a13aec6cad04092a51e094fc279f8f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a13aec6cad04092a51e094fc279f8f1, State: Initialized, Role: FOLLOWER
I20260806 01:02:25.324263 13028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a13aec6cad04092a51e094fc279f8f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a13aec6cad04092a51e094fc279f8f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } }
I20260806 01:02:25.324417 13028 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a13aec6cad04092a51e094fc279f8f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:02:25.324471 13028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a13aec6cad04092a51e094fc279f8f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:02:25.324535 13028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a13aec6cad04092a51e094fc279f8f1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:25.325271 13028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a13aec6cad04092a51e094fc279f8f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a13aec6cad04092a51e094fc279f8f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } }
I20260806 01:02:25.325409 13028 leader_election.cc:302] T 00000000000000000000000000000000 P 0a13aec6cad04092a51e094fc279f8f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a13aec6cad04092a51e094fc279f8f1; no voters:
I20260806 01:02:25.325613 13028 leader_election.cc:288] T 00000000000000000000000000000000 P 0a13aec6cad04092a51e094fc279f8f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:02:25.325696 13034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a13aec6cad04092a51e094fc279f8f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:25.326047 13034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a13aec6cad04092a51e094fc279f8f1 [term 1 LEADER]: Becoming Leader. State: Replica: 0a13aec6cad04092a51e094fc279f8f1, State: Running, Role: LEADER
I20260806 01:02:25.326076 13028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a13aec6cad04092a51e094fc279f8f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:25.326175 13034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a13aec6cad04092a51e094fc279f8f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a13aec6cad04092a51e094fc279f8f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } }
I20260806 01:02:25.327358 13036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a13aec6cad04092a51e094fc279f8f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a13aec6cad04092a51e094fc279f8f1. Latest consensus state: current_term: 1 leader_uuid: "0a13aec6cad04092a51e094fc279f8f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a13aec6cad04092a51e094fc279f8f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } } }
I20260806 01:02:25.327508 13036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a13aec6cad04092a51e094fc279f8f1 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:25.327775 13035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a13aec6cad04092a51e094fc279f8f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a13aec6cad04092a51e094fc279f8f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a13aec6cad04092a51e094fc279f8f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33481 } } }
I20260806 01:02:25.327888 13035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a13aec6cad04092a51e094fc279f8f1 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:25.327821 13043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:02:25.328812 13043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:02:25.330474 13032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:25.317394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48653 (local address 127.0.0.1:33481)
0806 01:02:25.317643 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:25.317653 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:25.317689 (+ 36us) server_negotiation.cc:407] Connection header received
0806 01:02:25.318419 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:25.318428 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:25.318490 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:25.318703 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:25.319178 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:25.321193 (+ 2015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:25.322383 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:25.323585 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:25.323775 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:25.329438 (+ 5663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:25.329444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:25.329444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:25.329750 (+ 306us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:25.329808 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:25.329826 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:25.329938 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:25.330216 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 01:02:25.330256 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":114,"threads_started":1}
I20260806 01:02:25.331614 13043 catalog_manager.cc:1347] Generated new cluster ID: b4e9c44fd6c04c4ab7a0d86631273270
I20260806 01:02:25.331679 13043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:02:25.331979 24520 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 01:02:25.338408 13032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:25.333576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37475 (local address 127.0.0.1:33481)
0806 01:02:25.333614 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:25.333616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:25.333934 (+ 318us) server_negotiation.cc:407] Connection header received
0806 01:02:25.333964 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:25.333965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:25.333976 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:25.334050 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:25.334489 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:25.335295 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:25.336588 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:25.336939 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:25.337003 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:25.337998 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:25.337999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:25.338000 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:25.338006 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:25.338024 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:25.338028 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:25.338092 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:25.338297 (+ 205us) server_negotiation.cc:299] Negotiation successful
0806 01:02:25.338314 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 01:02:25.348074 13043 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:02:25.348729 13043 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:02:25.353849 13043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a13aec6cad04092a51e094fc279f8f1: Generated new TSK 0
I20260806 01:02:25.354229 13043 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 01:02:25.358177 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 12958
W20260806 01:02:25.364212 12943 connection.cc:535] client connection to 127.0.0.1:33481 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33481): Connection reset by peer (error 104)
2026-08-06T01:02:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1621 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 01:02:25 dist-test-slave-dhph krb5kdc[13062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 01:02:25 dist-test-slave-dhph krb5kdc[13062](info): set up 2 sockets
Aug 06 01:02:25 dist-test-slave-dhph krb5kdc[13062](info): commencing operation
krb5kdc: starting...
W20260806 01:02:26.565665 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.187s user 0.004s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 01:02:26 dist-test-slave-dhph krb5kdc[13062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T01:02:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:02:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 01:02:26.723589 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36991
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:26.821928 13075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:26.822216 13075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:26.822268 13075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:26.824047 13075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:26.824246 13075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:26.824286 13075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:26.824316 13075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:26.824517 13075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:26.826373 13075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36991
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:26.826732 13075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:26.827003 13075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:26.833299 13084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:26.833299 13082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:26.833314 13081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:26.833674 13075 server_base.cc:1034] running on GCE node
I20260806 01:02:26.833977 13075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:26.834293 13075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:26.835521 13075 hybrid_clock.cc:648] HybridClock initialized: now 1785978146835489 us; error 72 us; skew 500 ppm
Aug 06 01:02:26 dist-test-slave-dhph krb5kdc[13062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:26.838423 13075 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 01:02:26.838992 13075 webserver.cc:460] Webserver started at http://127.0.0.1:44661/ using document root <none> and password file <none>
I20260806 01:02:26.839283 13075 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:26.839337 13075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:26.839483 13075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:26.841266 13075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "eb492ac0f736474d912230cc6b75e8bf"
format_stamp: "Formatted at 2026-08-06 01:02:26 on dist-test-slave-dhph"
server_key: "03430755c4f52d53eb76c89136ea7a6e"
server_key_iv: "8397b20be39026cb66453174e4d1568e"
server_key_version: "encryptionkey@0"
I20260806 01:02:26.841943 13075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "eb492ac0f736474d912230cc6b75e8bf"
format_stamp: "Formatted at 2026-08-06 01:02:26 on dist-test-slave-dhph"
server_key: "03430755c4f52d53eb76c89136ea7a6e"
server_key_iv: "8397b20be39026cb66453174e4d1568e"
server_key_version: "encryptionkey@0"
I20260806 01:02:26.844329 13075 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.003s
I20260806 01:02:26.845477 13091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:26.845801 13075 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:26.845943 13075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "eb492ac0f736474d912230cc6b75e8bf"
format_stamp: "Formatted at 2026-08-06 01:02:26 on dist-test-slave-dhph"
server_key: "03430755c4f52d53eb76c89136ea7a6e"
server_key_iv: "8397b20be39026cb66453174e4d1568e"
server_key_version: "encryptionkey@0"
I20260806 01:02:26.846036 13075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:26.849730 13075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:26.854342 13075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:26.854714 13075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:26.861203 13075 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36991
I20260806 01:02:26.861282 13146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36991 every 8 connection(s)
I20260806 01:02:26.862264 13075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:02:26.863094 13147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:26.865494 13147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb492ac0f736474d912230cc6b75e8bf: Bootstrap starting.
I20260806 01:02:26.866128 13147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb492ac0f736474d912230cc6b75e8bf: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:26.866464 13147 log.cc:824] T 00000000000000000000000000000000 P eb492ac0f736474d912230cc6b75e8bf: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:26.867347 13147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb492ac0f736474d912230cc6b75e8bf: No bootstrap required, opened a new log
I20260806 01:02:26.869736 13147 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb492ac0f736474d912230cc6b75e8bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb492ac0f736474d912230cc6b75e8bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36991 } }
I20260806 01:02:26.869896 13147 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb492ac0f736474d912230cc6b75e8bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:26.869920 13147 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb492ac0f736474d912230cc6b75e8bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb492ac0f736474d912230cc6b75e8bf, State: Initialized, Role: FOLLOWER
I20260806 01:02:26.870090 13147 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb492ac0f736474d912230cc6b75e8bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb492ac0f736474d912230cc6b75e8bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36991 } }
I20260806 01:02:26.870172 13147 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb492ac0f736474d912230cc6b75e8bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:02:26.870194 13147 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb492ac0f736474d912230cc6b75e8bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:02:26.870256 13147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb492ac0f736474d912230cc6b75e8bf [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:26.870990 13147 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb492ac0f736474d912230cc6b75e8bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb492ac0f736474d912230cc6b75e8bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36991 } }
I20260806 01:02:26.871090 13147 leader_election.cc:302] T 00000000000000000000000000000000 P eb492ac0f736474d912230cc6b75e8bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb492ac0f736474d912230cc6b75e8bf; no voters:
I20260806 01:02:26.871290 13147 leader_election.cc:288] T 00000000000000000000000000000000 P eb492ac0f736474d912230cc6b75e8bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:02:26.871349 13150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb492ac0f736474d912230cc6b75e8bf [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:26.871477 13150 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb492ac0f736474d912230cc6b75e8bf [term 1 LEADER]: Becoming Leader. State: Replica: eb492ac0f736474d912230cc6b75e8bf, State: Running, Role: LEADER
I20260806 01:02:26.871596 13147 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb492ac0f736474d912230cc6b75e8bf [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:26.871658 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 13075
I20260806 01:02:26.871873 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:02:26.871603 13150 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb492ac0f736474d912230cc6b75e8bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb492ac0f736474d912230cc6b75e8bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36991 } }
I20260806 01:02:26.872125 24520 external_mini_cluster.cc:1428] Setting key 29692d7feedf0779c15ce2bb1cc05044
I20260806 01:02:26.873278 13151 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb492ac0f736474d912230cc6b75e8bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb492ac0f736474d912230cc6b75e8bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb492ac0f736474d912230cc6b75e8bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36991 } } }
I20260806 01:02:26.873396 13151 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb492ac0f736474d912230cc6b75e8bf [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:26.873649 13152 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb492ac0f736474d912230cc6b75e8bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb492ac0f736474d912230cc6b75e8bf. Latest consensus state: current_term: 1 leader_uuid: "eb492ac0f736474d912230cc6b75e8bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb492ac0f736474d912230cc6b75e8bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36991 } } }
I20260806 01:02:26.873871 13162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:02:26.873966 13152 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb492ac0f736474d912230cc6b75e8bf [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:26.875074 13162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:02:26.875149 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:26.872788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51067 (local address 127.0.0.1:36991)
0806 01:02:26.873051 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:26.873062 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:26.873099 (+ 37us) server_negotiation.cc:407] Connection header received
0806 01:02:26.873444 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:26.873455 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:26.873503 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:26.874063 (+ 560us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:26.874564 (+ 501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:26.874568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:26.874568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:26.874572 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:26.874649 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:26.874666 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:26.874748 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:26.874883 (+ 135us) server_negotiation.cc:299] Negotiation successful
0806 01:02:26.874931 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":149,"threads_started":1}
I20260806 01:02:26.877187 24520 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 01:02:26.878659 13162 catalog_manager.cc:1347] Generated new cluster ID: a338d71be0de42f7a44a26ad6e68b23e
I20260806 01:02:26.878815 13162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:02:26.885075 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:26.883368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45695 (local address 127.0.0.1:36991)
0806 01:02:26.883417 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:26.883419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:26.883898 (+ 479us) server_negotiation.cc:407] Connection header received
0806 01:02:26.883962 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:26.883964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:26.883971 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:26.884042 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:26.884791 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:26.884792 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:26.884793 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:26.884797 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:26.884816 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:26.884821 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:26.884885 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:26.884960 (+ 75us) server_negotiation.cc:299] Negotiation successful
0806 01:02:26.884975 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 01:02:26.911010 13162 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:02:26.911242 13162 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:02:26.921290 13162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb492ac0f736474d912230cc6b75e8bf: Generated new TSK 0
I20260806 01:02:26.922012 13162 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 01:02:26.928623 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 13075
2026-08-06T01:02:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1572 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 01:02:27 dist-test-slave-dhph krb5kdc[13180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 01:02:27 dist-test-slave-dhph krb5kdc[13180](info): set up 2 sockets
Aug 06 01:02:27 dist-test-slave-dhph krb5kdc[13180](info): commencing operation
krb5kdc: starting...
W20260806 01:02:28.087210 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 01:02:28 dist-test-slave-dhph krb5kdc[13180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T01:02:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:02:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 01:02:28.243139 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39347
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:28.341037 13194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:28.341341 13194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:28.341390 13194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:28.343832 13194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:28.343958 13194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:28.343991 13194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:28.344017 13194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:28.344048 13194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:28.346057 13194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:28.346431 13194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:28.347013 13194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:28.353637 13199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:28.353682 13200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:28.353735 13194 server_base.cc:1034] running on GCE node
W20260806 01:02:28.353650 13202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:28.354149 13194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:28.354542 13194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:28.355813 13194 hybrid_clock.cc:648] HybridClock initialized: now 1785978148355781 us; error 42 us; skew 500 ppm
Aug 06 01:02:28 dist-test-slave-dhph krb5kdc[13180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:28.359297 13194 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 01:02:28.359879 13194 webserver.cc:460] Webserver started at http://127.0.0.1:42173/ using document root <none> and password file <none>
I20260806 01:02:28.360165 13194 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:28.360241 13194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:28.360407 13194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:28.361568 13194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "0cf79f9748cd4934844df7aa9f94ef3c"
format_stamp: "Formatted at 2026-08-06 01:02:28 on dist-test-slave-dhph"
server_key: "083fecdcec89cfcf0555c8c29e681aa1"
server_key_iv: "5c54ea4fa75905179da2e43b9a187a74"
server_key_version: "encryptionkey@0"
I20260806 01:02:28.361976 13194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "0cf79f9748cd4934844df7aa9f94ef3c"
format_stamp: "Formatted at 2026-08-06 01:02:28 on dist-test-slave-dhph"
server_key: "083fecdcec89cfcf0555c8c29e681aa1"
server_key_iv: "5c54ea4fa75905179da2e43b9a187a74"
server_key_version: "encryptionkey@0"
I20260806 01:02:28.363741 13194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 01:02:28.364780 13209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:28.365197 13194 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:28.365343 13194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "0cf79f9748cd4934844df7aa9f94ef3c"
format_stamp: "Formatted at 2026-08-06 01:02:28 on dist-test-slave-dhph"
server_key: "083fecdcec89cfcf0555c8c29e681aa1"
server_key_iv: "5c54ea4fa75905179da2e43b9a187a74"
server_key_version: "encryptionkey@0"
I20260806 01:02:28.365442 13194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:28.390221 13194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:28.393535 13194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:28.393801 13194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:28.400162 13194 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39347
I20260806 01:02:28.400280 13262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39347 every 8 connection(s)
I20260806 01:02:28.400691 13194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:02:28.401700 13263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:28.404210 13263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf79f9748cd4934844df7aa9f94ef3c: Bootstrap starting.
I20260806 01:02:28.404845 13263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cf79f9748cd4934844df7aa9f94ef3c: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:28.405156 13263 log.cc:824] T 00000000000000000000000000000000 P 0cf79f9748cd4934844df7aa9f94ef3c: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:28.406117 13263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cf79f9748cd4934844df7aa9f94ef3c: No bootstrap required, opened a new log
I20260806 01:02:28.408043 13263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cf79f9748cd4934844df7aa9f94ef3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf79f9748cd4934844df7aa9f94ef3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39347 } }
I20260806 01:02:28.408196 13263 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cf79f9748cd4934844df7aa9f94ef3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:28.408221 13263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cf79f9748cd4934844df7aa9f94ef3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf79f9748cd4934844df7aa9f94ef3c, State: Initialized, Role: FOLLOWER
I20260806 01:02:28.408388 13263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cf79f9748cd4934844df7aa9f94ef3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf79f9748cd4934844df7aa9f94ef3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39347 } }
I20260806 01:02:28.408456 13263 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cf79f9748cd4934844df7aa9f94ef3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:02:28.408479 13263 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cf79f9748cd4934844df7aa9f94ef3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:02:28.408531 13263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cf79f9748cd4934844df7aa9f94ef3c [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:28.409185 13263 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cf79f9748cd4934844df7aa9f94ef3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf79f9748cd4934844df7aa9f94ef3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39347 } }
I20260806 01:02:28.409287 13263 leader_election.cc:302] T 00000000000000000000000000000000 P 0cf79f9748cd4934844df7aa9f94ef3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cf79f9748cd4934844df7aa9f94ef3c; no voters:
I20260806 01:02:28.409485 13263 leader_election.cc:288] T 00000000000000000000000000000000 P 0cf79f9748cd4934844df7aa9f94ef3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:02:28.409722 13267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cf79f9748cd4934844df7aa9f94ef3c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:28.410059 13267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cf79f9748cd4934844df7aa9f94ef3c [term 1 LEADER]: Becoming Leader. State: Replica: 0cf79f9748cd4934844df7aa9f94ef3c, State: Running, Role: LEADER
I20260806 01:02:28.410141 13263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cf79f9748cd4934844df7aa9f94ef3c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:28.410312 13267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cf79f9748cd4934844df7aa9f94ef3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf79f9748cd4934844df7aa9f94ef3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39347 } }
I20260806 01:02:28.410830 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 13194
I20260806 01:02:28.411008 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:02:28.411270 24520 external_mini_cluster.cc:1428] Setting key 2215c6f6c6a3e5e52f7fe2e8b442308b
I20260806 01:02:28.411425 13269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf79f9748cd4934844df7aa9f94ef3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cf79f9748cd4934844df7aa9f94ef3c. Latest consensus state: current_term: 1 leader_uuid: "0cf79f9748cd4934844df7aa9f94ef3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf79f9748cd4934844df7aa9f94ef3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39347 } } }
I20260806 01:02:28.411540 13269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf79f9748cd4934844df7aa9f94ef3c [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:28.411778 13268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cf79f9748cd4934844df7aa9f94ef3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cf79f9748cd4934844df7aa9f94ef3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cf79f9748cd4934844df7aa9f94ef3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39347 } } }
I20260806 01:02:28.412228 13268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cf79f9748cd4934844df7aa9f94ef3c [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:28.411908 13277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:02:28.413199 13277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:02:28.415171 13277 catalog_manager.cc:1347] Generated new cluster ID: 5d1697c6262041dd95154ca7df270a05
I20260806 01:02:28.415242 13277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:02:28.423229 13277 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:02:28.424278 13277 catalog_manager.cc:1504] Loading token signing keys...
Aug 06 01:02:28 dist-test-slave-dhph krb5kdc[13180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 01:02:28.429445 13277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cf79f9748cd4934844df7aa9f94ef3c: Generated new TSK 0
I20260806 01:02:28.429700 13277 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 01:02:28.430518 13282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:28.412128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60913 (local address 127.0.0.1:39347)
0806 01:02:28.413504 (+ 1376us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:28.413592 (+ 88us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:28.413664 (+ 72us) server_negotiation.cc:407] Connection header received
0806 01:02:28.413977 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:28.413987 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:28.414059 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:28.414235 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:28.415195 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:28.416328 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:28.417236 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:28.417717 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:28.417864 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:28.425399 (+ 7535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:28.425406 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:28.425407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:28.425426 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:28.427662 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:28.428397 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:28.428407 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:28.428407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:28.428795 (+ 388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:28.429207 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:28.429208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:28.429209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:28.429591 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:28.429769 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:28.430085 (+ 316us) server_negotiation.cc:299] Negotiation successful
0806 01:02:28.430287 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":173,"thread_start_us":84,"threads_started":1}
I20260806 01:02:28.432067 24520 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 01:02:28.444841 13282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:28.433967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49179 (local address 127.0.0.1:39347)
0806 01:02:28.434049 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:28.434052 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:28.434246 (+ 194us) server_negotiation.cc:407] Connection header received
0806 01:02:28.434404 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:28.434406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:28.434424 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:28.434552 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:28.435279 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:28.435999 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:28.436916 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:28.437174 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:28.437255 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:28.440135 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:28.440138 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:28.440138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:28.440158 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:28.443116 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:28.443816 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:28.443818 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:28.443819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:28.443891 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:28.444165 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:28.444166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:28.444166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:28.444304 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:28.444448 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:28.444587 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 01:02:28.444714 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 01:02:28.447851 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 13194
2026-08-06T01:02:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1517 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 01:02:28 dist-test-slave-dhph krb5kdc[13297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 01:02:28 dist-test-slave-dhph krb5kdc[13297](info): set up 2 sockets
Aug 06 01:02:28 dist-test-slave-dhph krb5kdc[13297](info): commencing operation
krb5kdc: starting...
W20260806 01:02:29.627048 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 01:02:29 dist-test-slave-dhph krb5kdc[13297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T01:02:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:02:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 01:02:29.783531 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43585
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:29.880125 13311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:29.880474 13311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:29.880509 13311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:29.882555 13311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:29.882725 13311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:29.882746 13311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:29.882763 13311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:29.882839 13311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:29.885421 13311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:29.885743 13311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:29.886010 13311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:29.891580 13316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:29.891567 13319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:29.891561 13317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:29.892088 13311 server_base.cc:1034] running on GCE node
I20260806 01:02:29.892467 13311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:29.892822 13311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:29.894025 13311 hybrid_clock.cc:648] HybridClock initialized: now 1785978149894002 us; error 41 us; skew 500 ppm
Aug 06 01:02:29 dist-test-slave-dhph krb5kdc[13297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:29.897572 13311 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 01:02:29.898231 13311 webserver.cc:460] Webserver started at http://127.0.0.1:34027/ using document root <none> and password file <none>
I20260806 01:02:29.898509 13311 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:29.898614 13311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:29.898797 13311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:29.900326 13311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "439fb63e88ff48a8a1f6fb5dbd7a747a"
format_stamp: "Formatted at 2026-08-06 01:02:29 on dist-test-slave-dhph"
server_key: "7ac7c9bb8e3ea4c284026013af75ba67"
server_key_iv: "b5ad43634f597b25f56f1d8741af0fc8"
server_key_version: "encryptionkey@0"
I20260806 01:02:29.900787 13311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "439fb63e88ff48a8a1f6fb5dbd7a747a"
format_stamp: "Formatted at 2026-08-06 01:02:29 on dist-test-slave-dhph"
server_key: "7ac7c9bb8e3ea4c284026013af75ba67"
server_key_iv: "b5ad43634f597b25f56f1d8741af0fc8"
server_key_version: "encryptionkey@0"
I20260806 01:02:29.902838 13311 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260806 01:02:29.903864 13327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:29.904250 13311 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:29.904484 13311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "439fb63e88ff48a8a1f6fb5dbd7a747a"
format_stamp: "Formatted at 2026-08-06 01:02:29 on dist-test-slave-dhph"
server_key: "7ac7c9bb8e3ea4c284026013af75ba67"
server_key_iv: "b5ad43634f597b25f56f1d8741af0fc8"
server_key_version: "encryptionkey@0"
I20260806 01:02:29.904614 13311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:29.919385 13311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:29.922183 13311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:29.922396 13311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:29.928850 13311 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43585
I20260806 01:02:29.928900 13383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43585 every 8 connection(s)
I20260806 01:02:29.929363 13311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:02:29.930436 13384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:29.930998 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 13311
I20260806 01:02:29.931138 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:02:29.931331 24520 external_mini_cluster.cc:1428] Setting key 50ede391a4148ee8ae284a39855f904d
I20260806 01:02:29.933679 13384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 439fb63e88ff48a8a1f6fb5dbd7a747a: Bootstrap starting.
I20260806 01:02:29.934481 13384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 439fb63e88ff48a8a1f6fb5dbd7a747a: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:29.934957 13384 log.cc:824] T 00000000000000000000000000000000 P 439fb63e88ff48a8a1f6fb5dbd7a747a: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:29.935806 13384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 439fb63e88ff48a8a1f6fb5dbd7a747a: No bootstrap required, opened a new log
I20260806 01:02:29.937747 13388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:29.932005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41533 (local address 127.0.0.1:43585)
0806 01:02:29.932210 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:29.932240 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:29.932270 (+ 30us) server_negotiation.cc:407] Connection header received
0806 01:02:29.932728 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:29.932735 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:29.932785 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:29.933117 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:29.933386 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:29.934693 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:29.935585 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:29.936062 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:29.936345 (+ 283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:29.937116 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:29.937123 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:29.937124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:29.937300 (+ 176us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:29.937331 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:29.937343 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:29.937415 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:29.937493 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 01:02:29.937563 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":100,"threads_started":1}
I20260806 01:02:29.938197 13384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 439fb63e88ff48a8a1f6fb5dbd7a747a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439fb63e88ff48a8a1f6fb5dbd7a747a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43585 } }
I20260806 01:02:29.938419 13384 raft_consensus.cc:374] T 00000000000000000000000000000000 P 439fb63e88ff48a8a1f6fb5dbd7a747a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:29.938477 13384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 439fb63e88ff48a8a1f6fb5dbd7a747a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 439fb63e88ff48a8a1f6fb5dbd7a747a, State: Initialized, Role: FOLLOWER
I20260806 01:02:29.938642 13384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 439fb63e88ff48a8a1f6fb5dbd7a747a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439fb63e88ff48a8a1f6fb5dbd7a747a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43585 } }
I20260806 01:02:29.938773 13384 raft_consensus.cc:388] T 00000000000000000000000000000000 P 439fb63e88ff48a8a1f6fb5dbd7a747a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:02:29.938822 13384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 439fb63e88ff48a8a1f6fb5dbd7a747a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:02:29.938858 24520 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 01:02:29.938875 13384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 439fb63e88ff48a8a1f6fb5dbd7a747a [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:29.939636 13384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 439fb63e88ff48a8a1f6fb5dbd7a747a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439fb63e88ff48a8a1f6fb5dbd7a747a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43585 } }
I20260806 01:02:29.939800 13384 leader_election.cc:302] T 00000000000000000000000000000000 P 439fb63e88ff48a8a1f6fb5dbd7a747a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 439fb63e88ff48a8a1f6fb5dbd7a747a; no voters:
I20260806 01:02:29.940017 13384 leader_election.cc:288] T 00000000000000000000000000000000 P 439fb63e88ff48a8a1f6fb5dbd7a747a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:02:29.940053 13390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 439fb63e88ff48a8a1f6fb5dbd7a747a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:29.940169 13390 raft_consensus.cc:686] T 00000000000000000000000000000000 P 439fb63e88ff48a8a1f6fb5dbd7a747a [term 1 LEADER]: Becoming Leader. State: Replica: 439fb63e88ff48a8a1f6fb5dbd7a747a, State: Running, Role: LEADER
I20260806 01:02:29.940266 13390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 439fb63e88ff48a8a1f6fb5dbd7a747a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439fb63e88ff48a8a1f6fb5dbd7a747a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43585 } }
I20260806 01:02:29.940299 13384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 439fb63e88ff48a8a1f6fb5dbd7a747a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:29.940829 13390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 439fb63e88ff48a8a1f6fb5dbd7a747a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 439fb63e88ff48a8a1f6fb5dbd7a747a. Latest consensus state: current_term: 1 leader_uuid: "439fb63e88ff48a8a1f6fb5dbd7a747a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439fb63e88ff48a8a1f6fb5dbd7a747a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43585 } } }
I20260806 01:02:29.940840 13392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 439fb63e88ff48a8a1f6fb5dbd7a747a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "439fb63e88ff48a8a1f6fb5dbd7a747a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "439fb63e88ff48a8a1f6fb5dbd7a747a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43585 } } }
I20260806 01:02:29.941100 13392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 439fb63e88ff48a8a1f6fb5dbd7a747a [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:29.940948 13390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 439fb63e88ff48a8a1f6fb5dbd7a747a [sys.catalog]: This master's current role is: LEADER
W20260806 01:02:29.944176 13410 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 439fb63e88ff48a8a1f6fb5dbd7a747a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 01:02:29.944257 13410 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 01:02:29.944310 13402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:02:29.944662 13402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:02:29.945552 13388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:29.940954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34523 (local address 127.0.0.1:43585)
0806 01:02:29.941035 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:29.941037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:29.941619 (+ 582us) server_negotiation.cc:407] Connection header received
0806 01:02:29.941733 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:29.941735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:29.941748 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:29.941781 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:29.941985 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:29.942930 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:29.943988 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:29.944218 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:29.944339 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:29.945272 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:29.945274 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:29.945274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:29.945280 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:29.945297 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:29.945303 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:29.945358 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:29.945428 (+ 70us) server_negotiation.cc:299] Negotiation successful
0806 01:02:29.945454 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 01:02:29.946954 13402 catalog_manager.cc:1347] Generated new cluster ID: 7f62097f18cf4df6b732fab91a1faa67
I20260806 01:02:29.947021 13402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:02:29.957549 13402 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:02:29.958367 13402 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:02:29.961695 13402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 439fb63e88ff48a8a1f6fb5dbd7a747a: Generated new TSK 0
I20260806 01:02:29.961901 13402 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 01:02:29.969785 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 13311
2026-08-06T01:02:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1521 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 01:02:30 dist-test-slave-dhph krb5kdc[13417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 01:02:30 dist-test-slave-dhph krb5kdc[13417](info): set up 2 sockets
Aug 06 01:02:30 dist-test-slave-dhph krb5kdc[13417](info): commencing operation
krb5kdc: starting...
W20260806 01:02:31.167289 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 01:02:31 dist-test-slave-dhph krb5kdc[13417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T01:02:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:02:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 01:02:31.315311 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37929
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:31.408784 13431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:31.409063 13431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:31.409109 13431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:31.410387 13431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:31.410462 13431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:31.410490 13431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:31.410521 13431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:31.410547 13431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:31.412667 13431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:31.413414 13431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:31.413782 13431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:31.421316 13437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:31.421316 13439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:31.421622 13431 server_base.cc:1034] running on GCE node
W20260806 01:02:31.421625 13436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:31.422343 13431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:31.422750 13431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:31.423918 13431 hybrid_clock.cc:648] HybridClock initialized: now 1785978151423833 us; error 100 us; skew 500 ppm
Aug 06 01:02:31 dist-test-slave-dhph krb5kdc[13417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:31.439960 13431 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 01:02:31.440507 13431 webserver.cc:460] Webserver started at http://127.0.0.1:34047/ using document root <none> and password file <none>
I20260806 01:02:31.440774 13431 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:31.440847 13431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:31.441260 13431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:31.442821 13431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "7f90d7c5d4d546219c367d4d369865d8"
format_stamp: "Formatted at 2026-08-06 01:02:31 on dist-test-slave-dhph"
server_key: "1fec836778f9a646d1e3c98d0b7c86a9"
server_key_iv: "210a04132c40d592d37169bb6dc6fce3"
server_key_version: "encryptionkey@0"
I20260806 01:02:31.443225 13431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "7f90d7c5d4d546219c367d4d369865d8"
format_stamp: "Formatted at 2026-08-06 01:02:31 on dist-test-slave-dhph"
server_key: "1fec836778f9a646d1e3c98d0b7c86a9"
server_key_iv: "210a04132c40d592d37169bb6dc6fce3"
server_key_version: "encryptionkey@0"
I20260806 01:02:31.444789 13431 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 01:02:31.445721 13446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:31.446034 13431 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:31.446161 13431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "7f90d7c5d4d546219c367d4d369865d8"
format_stamp: "Formatted at 2026-08-06 01:02:31 on dist-test-slave-dhph"
server_key: "1fec836778f9a646d1e3c98d0b7c86a9"
server_key_iv: "210a04132c40d592d37169bb6dc6fce3"
server_key_version: "encryptionkey@0"
I20260806 01:02:31.446273 13431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:31.449924 13431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:31.452698 13431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:31.452906 13431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:31.458740 13431 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37929
I20260806 01:02:31.458770 13501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37929 every 8 connection(s)
I20260806 01:02:31.459554 13431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:02:31.460592 13502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:31.462715 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 13431
I20260806 01:02:31.462857 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:02:31.463054 24520 external_mini_cluster.cc:1428] Setting key 35c6a94d52d38c6cfbc9e3a72156ac83
I20260806 01:02:31.463613 13502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f90d7c5d4d546219c367d4d369865d8: Bootstrap starting.
I20260806 01:02:31.464545 13502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f90d7c5d4d546219c367d4d369865d8: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:31.465068 13502 log.cc:824] T 00000000000000000000000000000000 P 7f90d7c5d4d546219c367d4d369865d8: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:31.465987 13502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f90d7c5d4d546219c367d4d369865d8: No bootstrap required, opened a new log
I20260806 01:02:31.466555 13505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:31.464194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52887 (local address 127.0.0.1:37929)
0806 01:02:31.464439 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:31.464447 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:31.464474 (+ 27us) server_negotiation.cc:407] Connection header received
0806 01:02:31.464987 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:31.464994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:31.465030 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:31.465154 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:31.466066 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:31.466069 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:31.466070 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:31.466074 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:31.466125 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:31.466142 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:31.466233 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:31.466313 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 01:02:31.466353 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":125,"threads_started":1}
I20260806 01:02:31.468187 24520 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 01:02:31.468511 13502 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f90d7c5d4d546219c367d4d369865d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f90d7c5d4d546219c367d4d369865d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37929 } }
I20260806 01:02:31.468710 13502 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f90d7c5d4d546219c367d4d369865d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:31.468751 13502 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f90d7c5d4d546219c367d4d369865d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f90d7c5d4d546219c367d4d369865d8, State: Initialized, Role: FOLLOWER
I20260806 01:02:31.469029 13502 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f90d7c5d4d546219c367d4d369865d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f90d7c5d4d546219c367d4d369865d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37929 } }
I20260806 01:02:31.469144 13502 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f90d7c5d4d546219c367d4d369865d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:02:31.469178 13502 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f90d7c5d4d546219c367d4d369865d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:02:31.469219 13502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f90d7c5d4d546219c367d4d369865d8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:31.470116 13502 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f90d7c5d4d546219c367d4d369865d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f90d7c5d4d546219c367d4d369865d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37929 } }
I20260806 01:02:31.470229 13502 leader_election.cc:302] T 00000000000000000000000000000000 P 7f90d7c5d4d546219c367d4d369865d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f90d7c5d4d546219c367d4d369865d8; no voters:
I20260806 01:02:31.470427 13502 leader_election.cc:288] T 00000000000000000000000000000000 P 7f90d7c5d4d546219c367d4d369865d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:02:31.470654 13513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f90d7c5d4d546219c367d4d369865d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:31.470736 13502 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f90d7c5d4d546219c367d4d369865d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:31.470793 13513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f90d7c5d4d546219c367d4d369865d8 [term 1 LEADER]: Becoming Leader. State: Replica: 7f90d7c5d4d546219c367d4d369865d8, State: Running, Role: LEADER
I20260806 01:02:31.470909 13513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f90d7c5d4d546219c367d4d369865d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f90d7c5d4d546219c367d4d369865d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37929 } }
I20260806 01:02:31.471333 13505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:31.469846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39917 (local address 127.0.0.1:37929)
0806 01:02:31.469906 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:31.469908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:31.470125 (+ 217us) server_negotiation.cc:407] Connection header received
0806 01:02:31.470156 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:31.470157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:31.470164 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:31.470269 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:31.471075 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:31.471076 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:31.471076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:31.471080 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:31.471094 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:31.471098 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:31.471152 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:31.471243 (+ 91us) server_negotiation.cc:299] Negotiation successful
0806 01:02:31.471261 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 01:02:31.472055 13512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f90d7c5d4d546219c367d4d369865d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f90d7c5d4d546219c367d4d369865d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f90d7c5d4d546219c367d4d369865d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37929 } } }
I20260806 01:02:31.472319 13512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f90d7c5d4d546219c367d4d369865d8 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:31.472335 13514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f90d7c5d4d546219c367d4d369865d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f90d7c5d4d546219c367d4d369865d8. Latest consensus state: current_term: 1 leader_uuid: "7f90d7c5d4d546219c367d4d369865d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f90d7c5d4d546219c367d4d369865d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37929 } } }
I20260806 01:02:31.472441 13514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f90d7c5d4d546219c367d4d369865d8 [sys.catalog]: This master's current role is: LEADER
W20260806 01:02:31.472563 13527 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f90d7c5d4d546219c367d4d369865d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 01:02:31.472620 13527 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 01:02:31.472747 13530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:02:31.472936 13530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:02:31.474488 13530 catalog_manager.cc:1347] Generated new cluster ID: b530b185511d470ebb72f826c105a6d1
I20260806 01:02:31.474550 13530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:02:31.496762 13530 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:02:31.496995 13530 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:02:31.504766 13530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f90d7c5d4d546219c367d4d369865d8: Generated new TSK 0
I20260806 01:02:31.505061 13530 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 01:02:31.511247 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 13431
2026-08-06T01:02:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1540 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 01:02:31 dist-test-slave-dhph krb5kdc[13535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 01:02:31 dist-test-slave-dhph krb5kdc[13535](info): set up 2 sockets
Aug 06 01:02:31 dist-test-slave-dhph krb5kdc[13535](info): commencing operation
krb5kdc: starting...
W20260806 01:02:32.641850 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 01:02:32 dist-test-slave-dhph krb5kdc[13535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T01:02:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:02:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 01:02:32.795818 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32997
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:32.890152 13548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:32.890446 13548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:32.890493 13548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:32.891949 13548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:32.892115 13548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:32.892144 13548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:32.892169 13548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:32.892192 13548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:32.895327 13548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:32.895725 13548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:32.896008 13548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:32.902940 13554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:32.902943 13557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:32.902957 13555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:32.903609 13548 server_base.cc:1034] running on GCE node
I20260806 01:02:32.903872 13548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:32.904199 13548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:32.905423 13548 hybrid_clock.cc:648] HybridClock initialized: now 1785978152905396 us; error 81 us; skew 500 ppm
Aug 06 01:02:32 dist-test-slave-dhph krb5kdc[13535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:32.909240 13548 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 01:02:32.909910 13548 webserver.cc:460] Webserver started at http://127.0.0.1:35437/ using document root <none> and password file <none>
I20260806 01:02:32.910188 13548 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:32.910286 13548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:32.910463 13548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:32.911849 13548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "568598a34c054682b90fe915ca6efdbc"
format_stamp: "Formatted at 2026-08-06 01:02:32 on dist-test-slave-dhph"
server_key: "9228e4a41a4793072ecbcaaf6a643f0c"
server_key_iv: "fa2f57a9fedd0ec7aefe497cff7a12e4"
server_key_version: "encryptionkey@0"
I20260806 01:02:32.912299 13548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "568598a34c054682b90fe915ca6efdbc"
format_stamp: "Formatted at 2026-08-06 01:02:32 on dist-test-slave-dhph"
server_key: "9228e4a41a4793072ecbcaaf6a643f0c"
server_key_iv: "fa2f57a9fedd0ec7aefe497cff7a12e4"
server_key_version: "encryptionkey@0"
I20260806 01:02:32.914155 13548 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 01:02:32.915804 13564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:32.916201 13548 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 01:02:32.916496 13548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "568598a34c054682b90fe915ca6efdbc"
format_stamp: "Formatted at 2026-08-06 01:02:32 on dist-test-slave-dhph"
server_key: "9228e4a41a4793072ecbcaaf6a643f0c"
server_key_iv: "fa2f57a9fedd0ec7aefe497cff7a12e4"
server_key_version: "encryptionkey@0"
I20260806 01:02:32.916658 13548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:32.932216 13548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:32.935556 13548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:32.935784 13548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:32.940608 13548 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32997
I20260806 01:02:32.940649 13617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32997 every 8 connection(s)
I20260806 01:02:32.941064 13548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:02:32.942005 13618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:32.943241 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 13548
I20260806 01:02:32.943495 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:02:32.943783 24520 external_mini_cluster.cc:1428] Setting key b802ce8e306db92d04e1e085404e1526
I20260806 01:02:32.945221 13618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 568598a34c054682b90fe915ca6efdbc: Bootstrap starting.
I20260806 01:02:32.945856 13618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 568598a34c054682b90fe915ca6efdbc: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:32.946154 13618 log.cc:824] T 00000000000000000000000000000000 P 568598a34c054682b90fe915ca6efdbc: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:32.946981 13618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 568598a34c054682b90fe915ca6efdbc: No bootstrap required, opened a new log
I20260806 01:02:32.948872 13618 raft_consensus.cc:348] T 00000000000000000000000000000000 P 568598a34c054682b90fe915ca6efdbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "568598a34c054682b90fe915ca6efdbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } }
I20260806 01:02:32.949051 13618 raft_consensus.cc:374] T 00000000000000000000000000000000 P 568598a34c054682b90fe915ca6efdbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:32.949100 13618 raft_consensus.cc:729] T 00000000000000000000000000000000 P 568598a34c054682b90fe915ca6efdbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 568598a34c054682b90fe915ca6efdbc, State: Initialized, Role: FOLLOWER
I20260806 01:02:32.949290 13618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 568598a34c054682b90fe915ca6efdbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "568598a34c054682b90fe915ca6efdbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } }
I20260806 01:02:32.949394 13618 raft_consensus.cc:388] T 00000000000000000000000000000000 P 568598a34c054682b90fe915ca6efdbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:02:32.949440 13618 raft_consensus.cc:482] T 00000000000000000000000000000000 P 568598a34c054682b90fe915ca6efdbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:02:32.949491 13618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 568598a34c054682b90fe915ca6efdbc [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:32.950124 13618 raft_consensus.cc:504] T 00000000000000000000000000000000 P 568598a34c054682b90fe915ca6efdbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "568598a34c054682b90fe915ca6efdbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } }
I20260806 01:02:32.950263 13618 leader_election.cc:302] T 00000000000000000000000000000000 P 568598a34c054682b90fe915ca6efdbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 568598a34c054682b90fe915ca6efdbc; no voters:
I20260806 01:02:32.950475 13618 leader_election.cc:288] T 00000000000000000000000000000000 P 568598a34c054682b90fe915ca6efdbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:02:32.950569 13624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 568598a34c054682b90fe915ca6efdbc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:32.950767 13624 raft_consensus.cc:686] T 00000000000000000000000000000000 P 568598a34c054682b90fe915ca6efdbc [term 1 LEADER]: Becoming Leader. State: Replica: 568598a34c054682b90fe915ca6efdbc, State: Running, Role: LEADER
I20260806 01:02:32.950793 13618 sys_catalog.cc:564] T 00000000000000000000000000000000 P 568598a34c054682b90fe915ca6efdbc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:32.950882 13624 consensus_queue.cc:237] T 00000000000000000000000000000000 P 568598a34c054682b90fe915ca6efdbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "568598a34c054682b90fe915ca6efdbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } }
I20260806 01:02:32.951299 13624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 568598a34c054682b90fe915ca6efdbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 568598a34c054682b90fe915ca6efdbc. Latest consensus state: current_term: 1 leader_uuid: "568598a34c054682b90fe915ca6efdbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "568598a34c054682b90fe915ca6efdbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } } }
I20260806 01:02:32.951382 13624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 568598a34c054682b90fe915ca6efdbc [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:32.951465 13625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 568598a34c054682b90fe915ca6efdbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "568598a34c054682b90fe915ca6efdbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "568598a34c054682b90fe915ca6efdbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } } }
I20260806 01:02:32.951618 13625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 568598a34c054682b90fe915ca6efdbc [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:32.952557 13628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 06 01:02:32 dist-test-slave-dhph krb5kdc[13535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 01:02:32.953226 13628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:02:32.956772 13628 catalog_manager.cc:1347] Generated new cluster ID: 9b5c0ad2a4c64560a817694cbf2434bf
I20260806 01:02:32.956856 13628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:02:32.958330 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:32.944371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:47473 (local address 127.0.0.1:32997)
0806 01:02:32.944589 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:32.944598 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:32.944635 (+ 37us) server_negotiation.cc:407] Connection header received
0806 01:02:32.945054 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:32.945060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:32.945106 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:32.945266 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:32.946138 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:32.947951 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:32.949041 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:32.949786 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:32.949949 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:32.953761 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:32.953767 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:32.953768 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:32.953789 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:32.956296 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:32.956908 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:32.956912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:32.956912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:32.957054 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:32.957324 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:32.957325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:32.957326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:32.957613 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:32.957838 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:32.957950 (+ 112us) server_negotiation.cc:299] Negotiation successful
0806 01:02:32.958067 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":94,"threads_started":1}
I20260806 01:02:32.965324 24520 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 01:02:32.978434 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:32.971240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:52551 (local address 127.0.0.1:32997)
0806 01:02:32.971300 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:32.971302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:32.971446 (+ 144us) server_negotiation.cc:407] Connection header received
0806 01:02:32.971533 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:32.971534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:32.971546 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:32.971606 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:32.972255 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:32.972830 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:32.973637 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:32.973920 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:32.974008 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:32.975467 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:32.975468 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:32.975469 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:32.975478 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:32.977071 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:32.977491 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:32.977493 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:32.977493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:32.977562 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:32.977819 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:32.977821 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:32.977821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:32.977951 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:32.978082 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:32.978224 (+ 142us) server_negotiation.cc:299] Negotiation successful
0806 01:02:32.978297 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 01:02:32.983644 13628 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:02:32.984365 13628 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:02:32.987779 13628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 568598a34c054682b90fe915ca6efdbc: Generated new TSK 0
I20260806 01:02:32.988268 13628 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 01:02:32.993419 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 13548
W20260806 01:02:33.000119 13533 connection.cc:535] client connection to 127.0.0.1:32997 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32997): Connection reset by peer (error 104)
2026-08-06T01:02:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1482 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 01:02:33 dist-test-slave-dhph krb5kdc[13653](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 01:02:33 dist-test-slave-dhph krb5kdc[13653](info): set up 2 sockets
Aug 06 01:02:33 dist-test-slave-dhph krb5kdc[13653](info): commencing operation
krb5kdc: starting...
W20260806 01:02:34.167361 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 01:02:34 dist-test-slave-dhph krb5kdc[13653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T01:02:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:02:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 01:02:34.343720 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40835
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:34.450816 13666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:34.451108 13666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:34.451155 13666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:34.452518 13666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:34.452606 13666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:34.452633 13666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:34.452662 13666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:34.452695 13666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:34.454324 13666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:34.454663 13666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:34.454921 13666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:34.461072 13674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:34.461083 13672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:34.461145 13666 server_base.cc:1034] running on GCE node
W20260806 01:02:34.461145 13671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:34.462266 13666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:34.464035 13666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:34.465310 13666 hybrid_clock.cc:648] HybridClock initialized: now 1785978154465279 us; error 50 us; skew 500 ppm
Aug 06 01:02:34 dist-test-slave-dhph krb5kdc[13653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:34.468751 13666 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 01:02:34.469475 13666 webserver.cc:460] Webserver started at http://127.0.0.1:35533/ using document root <none> and password file <none>
I20260806 01:02:34.469722 13666 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:34.469777 13666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:34.469889 13666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:34.471191 13666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "bda5191c27cf41b2975c8487ce5012c3"
format_stamp: "Formatted at 2026-08-06 01:02:34 on dist-test-slave-dhph"
server_key: "474b1afef1ba3029ded5aeed754ff542"
server_key_iv: "6a68f464ec5585773197a0200e6acf30"
server_key_version: "encryptionkey@0"
I20260806 01:02:34.471674 13666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "bda5191c27cf41b2975c8487ce5012c3"
format_stamp: "Formatted at 2026-08-06 01:02:34 on dist-test-slave-dhph"
server_key: "474b1afef1ba3029ded5aeed754ff542"
server_key_iv: "6a68f464ec5585773197a0200e6acf30"
server_key_version: "encryptionkey@0"
I20260806 01:02:34.473167 13666 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 01:02:34.474215 13681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:34.474529 13666 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:34.474697 13666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "bda5191c27cf41b2975c8487ce5012c3"
format_stamp: "Formatted at 2026-08-06 01:02:34 on dist-test-slave-dhph"
server_key: "474b1afef1ba3029ded5aeed754ff542"
server_key_iv: "6a68f464ec5585773197a0200e6acf30"
server_key_version: "encryptionkey@0"
I20260806 01:02:34.474789 13666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:34.488731 13666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:34.492199 13666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:34.492405 13666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:34.497174 13666 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40835
I20260806 01:02:34.497327 13736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40835 every 8 connection(s)
I20260806 01:02:34.497663 13666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:02:34.498840 13737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:34.501233 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 13666
I20260806 01:02:34.501346 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:02:34.501546 24520 external_mini_cluster.cc:1428] Setting key 6d6130d4db901a03f4ff84c75f65df68
I20260806 01:02:34.501922 13737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda5191c27cf41b2975c8487ce5012c3: Bootstrap starting.
I20260806 01:02:34.502696 13737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bda5191c27cf41b2975c8487ce5012c3: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:34.503261 13737 log.cc:824] T 00000000000000000000000000000000 P bda5191c27cf41b2975c8487ce5012c3: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:34.505167 13737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda5191c27cf41b2975c8487ce5012c3: No bootstrap required, opened a new log
I20260806 01:02:34.508723 13737 raft_consensus.cc:348] T 00000000000000000000000000000000 P bda5191c27cf41b2975c8487ce5012c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda5191c27cf41b2975c8487ce5012c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40835 } }
I20260806 01:02:34.508993 13737 raft_consensus.cc:374] T 00000000000000000000000000000000 P bda5191c27cf41b2975c8487ce5012c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:34.509044 13737 raft_consensus.cc:729] T 00000000000000000000000000000000 P bda5191c27cf41b2975c8487ce5012c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda5191c27cf41b2975c8487ce5012c3, State: Initialized, Role: FOLLOWER
I20260806 01:02:34.509246 13737 consensus_queue.cc:260] T 00000000000000000000000000000000 P bda5191c27cf41b2975c8487ce5012c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda5191c27cf41b2975c8487ce5012c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40835 } }
I20260806 01:02:34.509353 13737 raft_consensus.cc:388] T 00000000000000000000000000000000 P bda5191c27cf41b2975c8487ce5012c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:02:34.509408 13737 raft_consensus.cc:482] T 00000000000000000000000000000000 P bda5191c27cf41b2975c8487ce5012c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:02:34.509784 13737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bda5191c27cf41b2975c8487ce5012c3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:34.510730 13737 raft_consensus.cc:504] T 00000000000000000000000000000000 P bda5191c27cf41b2975c8487ce5012c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda5191c27cf41b2975c8487ce5012c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40835 } }
I20260806 01:02:34.510877 13737 leader_election.cc:302] T 00000000000000000000000000000000 P bda5191c27cf41b2975c8487ce5012c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bda5191c27cf41b2975c8487ce5012c3; no voters:
I20260806 01:02:34.511041 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:34.502725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:47359 (local address 127.0.0.1:40835)
0806 01:02:34.502996 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:34.503009 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:34.503368 (+ 359us) server_negotiation.cc:407] Connection header received
0806 01:02:34.503940 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:34.503948 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:34.504001 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:34.504135 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:34.504727 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:34.506374 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:34.507738 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:34.508917 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:34.509079 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:34.510026 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:34.510031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:34.510031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:34.510225 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:34.510304 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:34.510318 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:34.510423 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:34.510674 (+ 251us) server_negotiation.cc:299] Negotiation successful
0806 01:02:34.510711 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":102,"threads_started":1}
I20260806 01:02:34.511148 13737 leader_election.cc:288] T 00000000000000000000000000000000 P bda5191c27cf41b2975c8487ce5012c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:02:34.511351 13745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bda5191c27cf41b2975c8487ce5012c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:34.511461 13737 sys_catalog.cc:564] T 00000000000000000000000000000000 P bda5191c27cf41b2975c8487ce5012c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:34.511512 13745 raft_consensus.cc:686] T 00000000000000000000000000000000 P bda5191c27cf41b2975c8487ce5012c3 [term 1 LEADER]: Becoming Leader. State: Replica: bda5191c27cf41b2975c8487ce5012c3, State: Running, Role: LEADER
I20260806 01:02:34.511704 13745 consensus_queue.cc:237] T 00000000000000000000000000000000 P bda5191c27cf41b2975c8487ce5012c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda5191c27cf41b2975c8487ce5012c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40835 } }
I20260806 01:02:34.512219 24520 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 01:02:34.512184 13744 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda5191c27cf41b2975c8487ce5012c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bda5191c27cf41b2975c8487ce5012c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda5191c27cf41b2975c8487ce5012c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40835 } } }
I20260806 01:02:34.512270 13744 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda5191c27cf41b2975c8487ce5012c3 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:34.512238 13745 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda5191c27cf41b2975c8487ce5012c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bda5191c27cf41b2975c8487ce5012c3. Latest consensus state: current_term: 1 leader_uuid: "bda5191c27cf41b2975c8487ce5012c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda5191c27cf41b2975c8487ce5012c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40835 } } }
I20260806 01:02:34.512318 13745 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda5191c27cf41b2975c8487ce5012c3 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:34.512473 13753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:02:34.512981 13753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:02:34.514809 13753 catalog_manager.cc:1347] Generated new cluster ID: 3bea8fbe4151477f8e25d1fae11b79ed
I20260806 01:02:34.514887 13753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:02:34.518023 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:34.514001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:54231 (local address 127.0.0.1:40835)
0806 01:02:34.514044 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:34.514046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:34.514228 (+ 182us) server_negotiation.cc:407] Connection header received
0806 01:02:34.514332 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:34.514332 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:34.514343 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:34.514396 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:34.514607 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:34.515454 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:34.516494 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:34.516851 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:34.516938 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:34.517737 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:34.517738 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:34.517739 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:34.517744 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:34.517762 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:34.517768 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:34.517834 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:34.517917 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 01:02:34.517929 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 01:02:34.529157 13753 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:02:34.530033 13753 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:02:34.533231 13753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bda5191c27cf41b2975c8487ce5012c3: Generated new TSK 0
I20260806 01:02:34.533501 13753 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 01:02:34.536763 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 13666
W20260806 01:02:34.543738 13651 connection.cc:535] client connection to 127.0.0.1:40835 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40835): Connection reset by peer (error 104)
2026-08-06T01:02:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1544 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 01:02:34 dist-test-slave-dhph krb5kdc[13771](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 01:02:34 dist-test-slave-dhph krb5kdc[13771](info): set up 2 sockets
Aug 06 01:02:34 dist-test-slave-dhph krb5kdc[13771](info): commencing operation
krb5kdc: starting...
W20260806 01:02:35.764421 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 01:02:35 dist-test-slave-dhph krb5kdc[13771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T01:02:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:02:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 01:02:35.952077 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38007
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:36.071074 13784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:36.071326 13784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:36.071359 13784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:36.072850 13784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:36.072916 13784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:36.072934 13784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:36.072949 13784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:36.072962 13784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:36.074855 13784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:36.075201 13784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:36.075515 13784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:36.081499 13792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:36.081496 13790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:36.081554 13789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:36.081526 13784 server_base.cc:1034] running on GCE node
I20260806 01:02:36.082018 13784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:36.082453 13784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:36.083683 13784 hybrid_clock.cc:648] HybridClock initialized: now 1785978156083651 us; error 54 us; skew 500 ppm
Aug 06 01:02:36 dist-test-slave-dhph krb5kdc[13771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:36.086740 13784 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 01:02:36.087303 13784 webserver.cc:460] Webserver started at http://127.0.0.1:42781/ using document root <none> and password file <none>
I20260806 01:02:36.087563 13784 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:36.087633 13784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:36.087790 13784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:36.088900 13784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "9ad21837881a41b6aa99d3a4fc522c95"
format_stamp: "Formatted at 2026-08-06 01:02:36 on dist-test-slave-dhph"
server_key: "3227c55ec606f76fcc4b25bfd2194d60"
server_key_iv: "8faaf3ccb4b349c2cf90292c3624e24a"
server_key_version: "encryptionkey@0"
I20260806 01:02:36.089311 13784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "9ad21837881a41b6aa99d3a4fc522c95"
format_stamp: "Formatted at 2026-08-06 01:02:36 on dist-test-slave-dhph"
server_key: "3227c55ec606f76fcc4b25bfd2194d60"
server_key_iv: "8faaf3ccb4b349c2cf90292c3624e24a"
server_key_version: "encryptionkey@0"
I20260806 01:02:36.091212 13784 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 01:02:36.092294 13799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:36.092773 13784 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:36.092918 13784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "9ad21837881a41b6aa99d3a4fc522c95"
format_stamp: "Formatted at 2026-08-06 01:02:36 on dist-test-slave-dhph"
server_key: "3227c55ec606f76fcc4b25bfd2194d60"
server_key_iv: "8faaf3ccb4b349c2cf90292c3624e24a"
server_key_version: "encryptionkey@0"
I20260806 01:02:36.093015 13784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:36.097249 13784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:36.100606 13784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:36.100997 13784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:36.106549 13784 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38007
I20260806 01:02:36.106724 13853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38007 every 8 connection(s)
I20260806 01:02:36.107259 13784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:02:36.108654 13854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:36.111418 13854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ad21837881a41b6aa99d3a4fc522c95: Bootstrap starting.
I20260806 01:02:36.111827 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 13784
I20260806 01:02:36.112004 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:02:36.112156 13854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ad21837881a41b6aa99d3a4fc522c95: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:36.112498 13854 log.cc:824] T 00000000000000000000000000000000 P 9ad21837881a41b6aa99d3a4fc522c95: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:36.112584 24520 external_mini_cluster.cc:1428] Setting key 180def74ec2cdd45e6610f95f833674a
I20260806 01:02:36.113534 13854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ad21837881a41b6aa99d3a4fc522c95: No bootstrap required, opened a new log
I20260806 01:02:36.116061 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:36.113278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:36141 (local address 127.0.0.1:38007)
0806 01:02:36.113505 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:36.113515 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:36.113549 (+ 34us) server_negotiation.cc:407] Connection header received
0806 01:02:36.114158 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:36.114168 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:36.114205 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:36.114369 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:36.115195 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:36.115200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:36.115200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:36.115205 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:36.115252 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:36.115266 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:36.115370 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:36.115430 (+ 60us) server_negotiation.cc:299] Negotiation successful
0806 01:02:36.115477 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":102,"threads_started":1}
I20260806 01:02:36.116436 13854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ad21837881a41b6aa99d3a4fc522c95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ad21837881a41b6aa99d3a4fc522c95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38007 } }
I20260806 01:02:36.116688 13854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ad21837881a41b6aa99d3a4fc522c95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:36.116742 13854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ad21837881a41b6aa99d3a4fc522c95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ad21837881a41b6aa99d3a4fc522c95, State: Initialized, Role: FOLLOWER
I20260806 01:02:36.116889 13854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ad21837881a41b6aa99d3a4fc522c95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ad21837881a41b6aa99d3a4fc522c95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38007 } }
I20260806 01:02:36.117010 13854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ad21837881a41b6aa99d3a4fc522c95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:02:36.117056 13854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ad21837881a41b6aa99d3a4fc522c95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:02:36.117115 13854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ad21837881a41b6aa99d3a4fc522c95 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:36.117642 24520 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 01:02:36.118023 13854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ad21837881a41b6aa99d3a4fc522c95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ad21837881a41b6aa99d3a4fc522c95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38007 } }
I20260806 01:02:36.118295 13854 leader_election.cc:302] T 00000000000000000000000000000000 P 9ad21837881a41b6aa99d3a4fc522c95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ad21837881a41b6aa99d3a4fc522c95; no voters:
I20260806 01:02:36.118693 13854 leader_election.cc:288] T 00000000000000000000000000000000 P 9ad21837881a41b6aa99d3a4fc522c95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:02:36.118832 13859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ad21837881a41b6aa99d3a4fc522c95 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:36.118974 13859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ad21837881a41b6aa99d3a4fc522c95 [term 1 LEADER]: Becoming Leader. State: Replica: 9ad21837881a41b6aa99d3a4fc522c95, State: Running, Role: LEADER
I20260806 01:02:36.119071 13854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ad21837881a41b6aa99d3a4fc522c95 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:36.119078 13859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ad21837881a41b6aa99d3a4fc522c95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ad21837881a41b6aa99d3a4fc522c95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38007 } }
I20260806 01:02:36.119647 13860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ad21837881a41b6aa99d3a4fc522c95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ad21837881a41b6aa99d3a4fc522c95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ad21837881a41b6aa99d3a4fc522c95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38007 } } }
I20260806 01:02:36.119660 13859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ad21837881a41b6aa99d3a4fc522c95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ad21837881a41b6aa99d3a4fc522c95. Latest consensus state: current_term: 1 leader_uuid: "9ad21837881a41b6aa99d3a4fc522c95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ad21837881a41b6aa99d3a4fc522c95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38007 } } }
I20260806 01:02:36.119987 13859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ad21837881a41b6aa99d3a4fc522c95 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:36.120031 13860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ad21837881a41b6aa99d3a4fc522c95 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:36.120226 13871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:02:36.120829 13871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:02:36.123850 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:36.119694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:48409 (local address 127.0.0.1:38007)
0806 01:02:36.119854 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:36.119863 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:36.122549 (+ 2686us) server_negotiation.cc:407] Connection header received
0806 01:02:36.122713 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:36.122714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:36.122722 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:36.122783 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:36.123587 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:36.123588 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:36.123588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:36.123593 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:36.123608 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:36.123613 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:36.123670 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:36.123757 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 01:02:36.123775 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 01:02:36.125021 13871 catalog_manager.cc:1347] Generated new cluster ID: 1bada5b5856c4740935959c452e9def7
I20260806 01:02:36.125077 13871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:02:36.136371 13871 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:02:36.136610 13871 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:02:36.145881 13871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ad21837881a41b6aa99d3a4fc522c95: Generated new TSK 0
I20260806 01:02:36.146126 13871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 01:02:36.149394 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 13784
2026-08-06T01:02:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1611 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 01:02:36 dist-test-slave-dhph krb5kdc[13887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 01:02:36 dist-test-slave-dhph krb5kdc[13887](info): set up 2 sockets
Aug 06 01:02:36 dist-test-slave-dhph krb5kdc[13887](info): commencing operation
krb5kdc: starting...
W20260806 01:02:37.314141 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.002s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 01:02:37 dist-test-slave-dhph krb5kdc[13887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T01:02:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:02:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 01:02:37.467710 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40129
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:37.562793 13901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:37.563015 13901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:37.563040 13901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:37.564231 13901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:37.564281 13901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:37.564296 13901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:37.564309 13901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:37.564321 13901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:37.565877 13901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40129
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:37.566149 13901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:37.566424 13901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:37.571642 13909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:37.571657 13906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:37.572002 13901 server_base.cc:1034] running on GCE node
W20260806 01:02:37.571929 13907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:37.572326 13901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:37.572643 13901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:37.573879 13901 hybrid_clock.cc:648] HybridClock initialized: now 1785978157573888 us; error 84 us; skew 500 ppm
Aug 06 01:02:37 dist-test-slave-dhph krb5kdc[13887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:37.584113 13901 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 01:02:37.584610 13901 webserver.cc:460] Webserver started at http://127.0.0.1:38561/ using document root <none> and password file <none>
I20260806 01:02:37.584877 13901 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:37.584947 13901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:37.585104 13901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:37.586228 13901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "bc852755df90475e9449f6dbfdeec36e"
format_stamp: "Formatted at 2026-08-06 01:02:37 on dist-test-slave-dhph"
server_key: "20d3ce3ffe07a83a880c499e79d5fa99"
server_key_iv: "23b6d869ae1bf0c35ef6765c9ff6858d"
server_key_version: "encryptionkey@0"
I20260806 01:02:37.586663 13901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "bc852755df90475e9449f6dbfdeec36e"
format_stamp: "Formatted at 2026-08-06 01:02:37 on dist-test-slave-dhph"
server_key: "20d3ce3ffe07a83a880c499e79d5fa99"
server_key_iv: "23b6d869ae1bf0c35ef6765c9ff6858d"
server_key_version: "encryptionkey@0"
I20260806 01:02:37.588387 13901 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260806 01:02:37.589382 13916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:37.589653 13901 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:37.589772 13901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "bc852755df90475e9449f6dbfdeec36e"
format_stamp: "Formatted at 2026-08-06 01:02:37 on dist-test-slave-dhph"
server_key: "20d3ce3ffe07a83a880c499e79d5fa99"
server_key_iv: "23b6d869ae1bf0c35ef6765c9ff6858d"
server_key_version: "encryptionkey@0"
I20260806 01:02:37.589866 13901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:37.616212 13901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:37.619822 13901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:37.620038 13901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:37.625906 13901 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40129
I20260806 01:02:37.625974 13971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40129 every 8 connection(s)
I20260806 01:02:37.626369 13901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:02:37.627445 13972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:37.630092 13972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc852755df90475e9449f6dbfdeec36e: Bootstrap starting.
I20260806 01:02:37.630868 13972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc852755df90475e9449f6dbfdeec36e: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:37.631201 13972 log.cc:824] T 00000000000000000000000000000000 P bc852755df90475e9449f6dbfdeec36e: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:37.632089 13972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc852755df90475e9449f6dbfdeec36e: No bootstrap required, opened a new log
I20260806 01:02:37.633956 13972 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc852755df90475e9449f6dbfdeec36e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc852755df90475e9449f6dbfdeec36e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40129 } }
I20260806 01:02:37.634119 13972 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc852755df90475e9449f6dbfdeec36e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:37.634147 13972 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc852755df90475e9449f6dbfdeec36e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc852755df90475e9449f6dbfdeec36e, State: Initialized, Role: FOLLOWER
I20260806 01:02:37.634335 13972 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc852755df90475e9449f6dbfdeec36e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc852755df90475e9449f6dbfdeec36e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40129 } }
I20260806 01:02:37.634408 13972 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc852755df90475e9449f6dbfdeec36e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:02:37.634433 13972 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc852755df90475e9449f6dbfdeec36e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:02:37.634488 13972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc852755df90475e9449f6dbfdeec36e [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:37.635205 13972 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc852755df90475e9449f6dbfdeec36e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc852755df90475e9449f6dbfdeec36e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40129 } }
I20260806 01:02:37.635308 13972 leader_election.cc:302] T 00000000000000000000000000000000 P bc852755df90475e9449f6dbfdeec36e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc852755df90475e9449f6dbfdeec36e; no voters:
I20260806 01:02:37.635540 13972 leader_election.cc:288] T 00000000000000000000000000000000 P bc852755df90475e9449f6dbfdeec36e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:02:37.635764 13975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc852755df90475e9449f6dbfdeec36e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:37.635918 13972 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc852755df90475e9449f6dbfdeec36e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:37.636112 13975 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc852755df90475e9449f6dbfdeec36e [term 1 LEADER]: Becoming Leader. State: Replica: bc852755df90475e9449f6dbfdeec36e, State: Running, Role: LEADER
I20260806 01:02:37.636186 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 13901
I20260806 01:02:37.636302 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:02:37.636277 13975 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc852755df90475e9449f6dbfdeec36e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc852755df90475e9449f6dbfdeec36e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40129 } }
I20260806 01:02:37.636629 24520 external_mini_cluster.cc:1428] Setting key 0af9e415d42d8210a22663b453ffd0b3
I20260806 01:02:37.637122 13976 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc852755df90475e9449f6dbfdeec36e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc852755df90475e9449f6dbfdeec36e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc852755df90475e9449f6dbfdeec36e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40129 } } }
I20260806 01:02:37.637142 13977 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc852755df90475e9449f6dbfdeec36e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc852755df90475e9449f6dbfdeec36e. Latest consensus state: current_term: 1 leader_uuid: "bc852755df90475e9449f6dbfdeec36e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc852755df90475e9449f6dbfdeec36e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40129 } } }
I20260806 01:02:37.637270 13976 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc852755df90475e9449f6dbfdeec36e [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:37.637290 13977 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc852755df90475e9449f6dbfdeec36e [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:37.637746 13985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:02:37.638415 13985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:02:37.640447 13985 catalog_manager.cc:1347] Generated new cluster ID: a53e4ac7f524405dbab139d499e5130e
I20260806 01:02:37.640516 13985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 01:02:37 dist-test-slave-dhph krb5kdc[13887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 01:02:37.652645 13985 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:02:37.653483 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:37.637552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:60341 (local address 127.0.0.1:40129)
0806 01:02:37.638132 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:37.638144 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:37.638193 (+ 49us) server_negotiation.cc:407] Connection header received
0806 01:02:37.638669 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:37.638679 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:37.638766 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:37.639742 (+ 976us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:37.640210 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:37.641875 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:37.643540 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:37.644195 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:37.644317 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:37.647574 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:37.647580 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:37.647580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:37.647597 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:37.649923 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:37.650816 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:37.650821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:37.650821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:37.650943 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:37.651218 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:37.651219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:37.651219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:37.652025 (+ 806us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:37.652513 (+ 488us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:37.652842 (+ 329us) server_negotiation.cc:299] Negotiation successful
0806 01:02:37.653144 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":93,"threads_started":1}
I20260806 01:02:37.653586 13985 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:02:37.654641 24520 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 01:02:37.659399 13985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc852755df90475e9449f6dbfdeec36e: Generated new TSK 0
I20260806 01:02:37.659627 13985 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 01:02:37.664727 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:37.656337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:43317 (local address 127.0.0.1:40129)
0806 01:02:37.656383 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:37.656385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:37.656395 (+ 10us) server_negotiation.cc:407] Connection header received
0806 01:02:37.656444 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:37.656445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:37.656455 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:37.656506 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:37.657104 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:37.657640 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:37.658847 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:37.659304 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:37.659399 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:37.661246 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:37.661248 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:37.661248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:37.661257 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:37.663133 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:37.663843 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:37.663845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:37.663845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:37.663920 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:37.664168 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:37.664169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:37.664169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:37.664300 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:37.664426 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:37.664551 (+ 125us) server_negotiation.cc:299] Negotiation successful
0806 01:02:37.664621 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 01:02:37.667552 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 13901
W20260806 01:02:37.674968 13885 connection.cc:535] client connection to 127.0.0.1:40129 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40129): Connection reset by peer (error 104)
2026-08-06T01:02:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1519 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 01:02:37 dist-test-slave-dhph krb5kdc[14004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 01:02:37 dist-test-slave-dhph krb5kdc[14004](info): set up 2 sockets
Aug 06 01:02:37 dist-test-slave-dhph krb5kdc[14004](info): commencing operation
krb5kdc: starting...
W20260806 01:02:38.812661 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 01:02:38 dist-test-slave-dhph krb5kdc[14004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T01:02:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:02:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 01:02:38.971442 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34049
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:39.064091 14020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:39.064452 14020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:39.064481 14020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:39.065802 14020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:39.065863 14020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:39.065878 14020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:39.065892 14020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:39.065905 14020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:39.067688 14020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34049
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:39.067978 14020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:39.068238 14020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:39.074038 14025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:39.074079 14026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:39.074038 14028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:39.074712 14020 server_base.cc:1034] running on GCE node
I20260806 01:02:39.074999 14020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:39.075336 14020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:39.076504 14020 hybrid_clock.cc:648] HybridClock initialized: now 1785978159076490 us; error 33 us; skew 500 ppm
Aug 06 01:02:39 dist-test-slave-dhph krb5kdc[14004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:39.086395 14020 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 01:02:39.087030 14020 webserver.cc:460] Webserver started at http://127.0.0.1:46691/ using document root <none> and password file <none>
I20260806 01:02:39.087340 14020 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:39.087396 14020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:39.087594 14020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:39.089308 14020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "2742557babdc404696c40385f0308c91"
format_stamp: "Formatted at 2026-08-06 01:02:39 on dist-test-slave-dhph"
server_key: "a55a948cc02f6cf7fc962b93a072a50c"
server_key_iv: "a2b24b5310b462423310d462b7885483"
server_key_version: "encryptionkey@0"
I20260806 01:02:39.089744 14020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "2742557babdc404696c40385f0308c91"
format_stamp: "Formatted at 2026-08-06 01:02:39 on dist-test-slave-dhph"
server_key: "a55a948cc02f6cf7fc962b93a072a50c"
server_key_iv: "a2b24b5310b462423310d462b7885483"
server_key_version: "encryptionkey@0"
I20260806 01:02:39.091439 14020 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260806 01:02:39.092518 14036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:39.092801 14020 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:39.092922 14020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "2742557babdc404696c40385f0308c91"
format_stamp: "Formatted at 2026-08-06 01:02:39 on dist-test-slave-dhph"
server_key: "a55a948cc02f6cf7fc962b93a072a50c"
server_key_iv: "a2b24b5310b462423310d462b7885483"
server_key_version: "encryptionkey@0"
I20260806 01:02:39.093004 14020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:39.116328 14020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:39.119498 14020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:39.119719 14020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:39.125020 14020 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34049
I20260806 01:02:39.125211 14091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34049 every 8 connection(s)
I20260806 01:02:39.125507 14020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:02:39.126513 14092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:39.128103 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 14020
I20260806 01:02:39.128342 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:02:39.128556 24520 external_mini_cluster.cc:1428] Setting key 8f70bea6ea0546ddd6bc01b98a588f26
I20260806 01:02:39.129585 14092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2742557babdc404696c40385f0308c91: Bootstrap starting.
I20260806 01:02:39.130398 14092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2742557babdc404696c40385f0308c91: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:39.130797 14092 log.cc:824] T 00000000000000000000000000000000 P 2742557babdc404696c40385f0308c91: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:39.131603 14092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2742557babdc404696c40385f0308c91: No bootstrap required, opened a new log
I20260806 01:02:39.133857 14092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2742557babdc404696c40385f0308c91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2742557babdc404696c40385f0308c91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34049 } }
I20260806 01:02:39.134081 14092 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2742557babdc404696c40385f0308c91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:39.134141 14092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2742557babdc404696c40385f0308c91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2742557babdc404696c40385f0308c91, State: Initialized, Role: FOLLOWER
I20260806 01:02:39.134351 14092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2742557babdc404696c40385f0308c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2742557babdc404696c40385f0308c91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34049 } }
I20260806 01:02:39.134466 14092 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2742557babdc404696c40385f0308c91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:02:39.134518 14092 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2742557babdc404696c40385f0308c91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:02:39.134572 14092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2742557babdc404696c40385f0308c91 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:39.135311 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:39.129180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:46253 (local address 127.0.0.1:34049)
0806 01:02:39.129354 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:39.129363 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:39.129391 (+ 28us) server_negotiation.cc:407] Connection header received
0806 01:02:39.129839 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:39.129846 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:39.129895 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:39.130014 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:39.130357 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:39.131605 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:39.132581 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:39.133104 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:39.133245 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:39.134565 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:39.134570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:39.134570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:39.134821 (+ 251us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:39.134858 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:39.134870 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:39.134977 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:39.135095 (+ 118us) server_negotiation.cc:299] Negotiation successful
0806 01:02:39.135126 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
I20260806 01:02:39.135454 14092 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2742557babdc404696c40385f0308c91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2742557babdc404696c40385f0308c91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34049 } }
I20260806 01:02:39.135583 14092 leader_election.cc:302] T 00000000000000000000000000000000 P 2742557babdc404696c40385f0308c91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2742557babdc404696c40385f0308c91; no voters:
I20260806 01:02:39.135802 14092 leader_election.cc:288] T 00000000000000000000000000000000 P 2742557babdc404696c40385f0308c91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:02:39.135850 14097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2742557babdc404696c40385f0308c91 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:39.136014 14097 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2742557babdc404696c40385f0308c91 [term 1 LEADER]: Becoming Leader. State: Replica: 2742557babdc404696c40385f0308c91, State: Running, Role: LEADER
I20260806 01:02:39.136137 14092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2742557babdc404696c40385f0308c91 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:39.136147 14097 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2742557babdc404696c40385f0308c91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2742557babdc404696c40385f0308c91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34049 } }
I20260806 01:02:39.136411 24520 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 01:02:39.136713 14098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2742557babdc404696c40385f0308c91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2742557babdc404696c40385f0308c91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2742557babdc404696c40385f0308c91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34049 } } }
I20260806 01:02:39.136801 14098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2742557babdc404696c40385f0308c91 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:39.137018 14099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2742557babdc404696c40385f0308c91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2742557babdc404696c40385f0308c91. Latest consensus state: current_term: 1 leader_uuid: "2742557babdc404696c40385f0308c91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2742557babdc404696c40385f0308c91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34049 } } }
I20260806 01:02:39.137307 14099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2742557babdc404696c40385f0308c91 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:39.137593 14104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:02:39.138273 14104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:02:39.140311 14104 catalog_manager.cc:1347] Generated new cluster ID: 38f0deafe4c243f5a7b15979ba3eaa3e
I20260806 01:02:39.140427 14104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:02:39.142819 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:39.138243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:35669 (local address 127.0.0.1:34049)
0806 01:02:39.138278 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:39.138280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:39.138600 (+ 320us) server_negotiation.cc:407] Connection header received
0806 01:02:39.138667 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:39.138668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:39.138679 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:39.138748 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:39.138937 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:39.139809 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:39.140932 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:39.141292 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:39.141401 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:39.142403 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:39.142405 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:39.142405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:39.142456 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:39.142476 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:39.142481 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:39.142559 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:39.142719 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 01:02:39.142738 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260806 01:02:39.179282 14104 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:02:39.179929 14104 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:02:39.183621 14104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2742557babdc404696c40385f0308c91: Generated new TSK 0
I20260806 01:02:39.183844 14104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 01:02:39.193645 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 14020
W20260806 01:02:39.200632 14002 connection.cc:535] client connection to 127.0.0.1:34049 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34049): Connection reset by peer (error 104)
2026-08-06T01:02:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1525 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 01:02:39 dist-test-slave-dhph krb5kdc[14125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 01:02:39 dist-test-slave-dhph krb5kdc[14125](info): set up 2 sockets
Aug 06 01:02:39 dist-test-slave-dhph krb5kdc[14125](info): commencing operation
krb5kdc: starting...
W20260806 01:02:40.373482 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 01:02:40 dist-test-slave-dhph krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T01:02:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:02:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 01:02:40.528364 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39417
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:40.631124 14139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:40.631353 14139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:40.631379 14139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:40.632541 14139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:40.632577 14139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:40.632591 14139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:40.632604 14139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:40.632617 14139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:40.634088 14139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39417
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785977869987768-24520-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:40.634320 14139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:40.634634 14139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:40.640229 14145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:40.640339 14144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:40.640434 14147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:40.641098 14139 server_base.cc:1034] running on GCE node
I20260806 01:02:40.641677 14139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:40.642004 14139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:40.643164 14139 hybrid_clock.cc:648] HybridClock initialized: now 1785978160643111 us; error 80 us; skew 500 ppm
Aug 06 01:02:40 dist-test-slave-dhph krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:40.646365 14139 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 01:02:40.647492 14139 webserver.cc:460] Webserver started at http://127.0.0.1:42997/ using document root <none> and password file <none>
I20260806 01:02:40.647750 14139 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:40.647797 14139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:40.647895 14139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:40.648975 14139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "f19e944cc1284c18a49e3c0501d179d6"
format_stamp: "Formatted at 2026-08-06 01:02:40 on dist-test-slave-dhph"
server_key: "805400d7a8214f1649416cf985fb2b2f"
server_key_iv: "020ba7ed493aca28f1e17d1c7848e882"
server_key_version: "encryptionkey@0"
I20260806 01:02:40.649284 14139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "f19e944cc1284c18a49e3c0501d179d6"
format_stamp: "Formatted at 2026-08-06 01:02:40 on dist-test-slave-dhph"
server_key: "805400d7a8214f1649416cf985fb2b2f"
server_key_iv: "020ba7ed493aca28f1e17d1c7848e882"
server_key_version: "encryptionkey@0"
I20260806 01:02:40.651019 14139 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260806 01:02:40.652103 14154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:40.652468 14139 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:40.652608 14139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "f19e944cc1284c18a49e3c0501d179d6"
format_stamp: "Formatted at 2026-08-06 01:02:40 on dist-test-slave-dhph"
server_key: "805400d7a8214f1649416cf985fb2b2f"
server_key_iv: "020ba7ed493aca28f1e17d1c7848e882"
server_key_version: "encryptionkey@0"
I20260806 01:02:40.652717 14139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:40.657100 14139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:40.660614 14139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:40.660774 14139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:40.665308 14139 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39417
I20260806 01:02:40.665508 14207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39417 every 8 connection(s)
I20260806 01:02:40.666108 14139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:02:40.667755 14208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:40.670854 14208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f19e944cc1284c18a49e3c0501d179d6: Bootstrap starting.
I20260806 01:02:40.671523 14208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f19e944cc1284c18a49e3c0501d179d6: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:40.671847 14208 log.cc:824] T 00000000000000000000000000000000 P f19e944cc1284c18a49e3c0501d179d6: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:40.672880 14208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f19e944cc1284c18a49e3c0501d179d6: No bootstrap required, opened a new log
I20260806 01:02:40.675278 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 14139
I20260806 01:02:40.675522 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:02:40.675772 24520 external_mini_cluster.cc:1428] Setting key aa7e2afd820b653c636b46d3afd10105
I20260806 01:02:40.675725 14208 raft_consensus.cc:348] T 00000000000000000000000000000000 P f19e944cc1284c18a49e3c0501d179d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19e944cc1284c18a49e3c0501d179d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39417 } }
I20260806 01:02:40.675921 14208 raft_consensus.cc:374] T 00000000000000000000000000000000 P f19e944cc1284c18a49e3c0501d179d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:40.675963 14208 raft_consensus.cc:729] T 00000000000000000000000000000000 P f19e944cc1284c18a49e3c0501d179d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19e944cc1284c18a49e3c0501d179d6, State: Initialized, Role: FOLLOWER
I20260806 01:02:40.676152 14208 consensus_queue.cc:260] T 00000000000000000000000000000000 P f19e944cc1284c18a49e3c0501d179d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19e944cc1284c18a49e3c0501d179d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39417 } }
I20260806 01:02:40.676342 14208 raft_consensus.cc:388] T 00000000000000000000000000000000 P f19e944cc1284c18a49e3c0501d179d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:02:40.676429 14208 raft_consensus.cc:482] T 00000000000000000000000000000000 P f19e944cc1284c18a49e3c0501d179d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:02:40.676517 14208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f19e944cc1284c18a49e3c0501d179d6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:40.678357 14208 raft_consensus.cc:504] T 00000000000000000000000000000000 P f19e944cc1284c18a49e3c0501d179d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19e944cc1284c18a49e3c0501d179d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39417 } }
I20260806 01:02:40.678508 14208 leader_election.cc:302] T 00000000000000000000000000000000 P f19e944cc1284c18a49e3c0501d179d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f19e944cc1284c18a49e3c0501d179d6; no voters:
I20260806 01:02:40.678752 14208 leader_election.cc:288] T 00000000000000000000000000000000 P f19e944cc1284c18a49e3c0501d179d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:02:40.678930 14214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f19e944cc1284c18a49e3c0501d179d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:40.679116 14214 raft_consensus.cc:686] T 00000000000000000000000000000000 P f19e944cc1284c18a49e3c0501d179d6 [term 1 LEADER]: Becoming Leader. State: Replica: f19e944cc1284c18a49e3c0501d179d6, State: Running, Role: LEADER
I20260806 01:02:40.679086 14208 sys_catalog.cc:564] T 00000000000000000000000000000000 P f19e944cc1284c18a49e3c0501d179d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:40.679217 14213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:40.676397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:39707 (local address 127.0.0.1:39417)
0806 01:02:40.676781 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:40.676792 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:40.676830 (+ 38us) server_negotiation.cc:407] Connection header received
0806 01:02:40.676959 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:40.676967 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:40.677010 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:40.677441 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:40.678323 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:40.678327 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:40.678327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:40.678332 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:40.678386 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:40.678402 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:40.678498 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:40.678730 (+ 232us) server_negotiation.cc:299] Negotiation successful
0806 01:02:40.678783 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":177,"threads_started":1}
I20260806 01:02:40.679267 14214 consensus_queue.cc:237] T 00000000000000000000000000000000 P f19e944cc1284c18a49e3c0501d179d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19e944cc1284c18a49e3c0501d179d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39417 } }
I20260806 01:02:40.679740 14216 sys_catalog.cc:455] T 00000000000000000000000000000000 P f19e944cc1284c18a49e3c0501d179d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f19e944cc1284c18a49e3c0501d179d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19e944cc1284c18a49e3c0501d179d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39417 } } }
I20260806 01:02:40.679770 14215 sys_catalog.cc:455] T 00000000000000000000000000000000 P f19e944cc1284c18a49e3c0501d179d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f19e944cc1284c18a49e3c0501d179d6. Latest consensus state: current_term: 1 leader_uuid: "f19e944cc1284c18a49e3c0501d179d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19e944cc1284c18a49e3c0501d179d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39417 } } }
I20260806 01:02:40.679880 14216 sys_catalog.cc:458] T 00000000000000000000000000000000 P f19e944cc1284c18a49e3c0501d179d6 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:40.679894 14215 sys_catalog.cc:458] T 00000000000000000000000000000000 P f19e944cc1284c18a49e3c0501d179d6 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:40.680445 24520 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 01:02:40.681078 14218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:02:40.682739 14218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:02:40.683715 14213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:40.682227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:56613 (local address 127.0.0.1:39417)
0806 01:02:40.682294 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:40.682298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:40.682402 (+ 104us) server_negotiation.cc:407] Connection header received
0806 01:02:40.682486 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:40.682487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:40.682496 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:40.682547 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:40.683464 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:40.683465 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:40.683466 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:40.683470 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:40.683486 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:40.683491 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:40.683541 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:40.683637 (+ 96us) server_negotiation.cc:299] Negotiation successful
0806 01:02:40.683654 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 01:02:40.684664 14218 catalog_manager.cc:1347] Generated new cluster ID: 5418d039cb8c4aefa68fdea3a48b1c80
I20260806 01:02:40.684707 14218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:02:40.691268 14218 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:02:40.691543 14218 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:02:40.701972 14218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f19e944cc1284c18a49e3c0501d179d6: Generated new TSK 0
I20260806 01:02:40.702270 14218 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 01:02:40.707461 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 14139
2026-08-06T01:02:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1512 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 01:02:40 dist-test-slave-dhph krb5kdc[14241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 01:02:40 dist-test-slave-dhph krb5kdc[14241](info): set up 2 sockets
Aug 06 01:02:40 dist-test-slave-dhph krb5kdc[14241](info): commencing operation
krb5kdc: starting...
W20260806 01:02:41.870834 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 01:02:41 dist-test-slave-dhph krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T01:02:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:02:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 01:02:42.035909 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:42113
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:33051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:42113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:42.138453 14255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:42.138826 14255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:42.138860 14255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:42.140233 14255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:42.140396 14255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:42.140440 14255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:42.140456 14255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:42.140470 14255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:42.142498 14255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:33051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:42113
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:42113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:42.142885 14255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:42.143205 14255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:42.149169 14263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:42.149169 14261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:42.149353 14260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:42.149501 14255 server_base.cc:1034] running on GCE node
I20260806 01:02:42.149950 14255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:42.150784 14255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:42.152374 14255 hybrid_clock.cc:648] HybridClock initialized: now 1785978162152311 us; error 77 us; skew 500 ppm
Aug 06 01:02:42 dist-test-slave-dhph krb5kdc[14241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978162, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:42.156061 14255 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 01:02:42.156793 14255 webserver.cc:460] Webserver started at http://127.23.242.62:37139/ using document root <none> and password file <none>
I20260806 01:02:42.157181 14255 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:42.157269 14255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:42.157431 14255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:42.158628 14255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "e206b800ff4140a494dfd4a520fee1f5"
format_stamp: "Formatted at 2026-08-06 01:02:42 on dist-test-slave-dhph"
server_key: "63cbc4c924de34a143f9f1a94d5f5599"
server_key_iv: "1f1de54bfb00a5cb34fdc79f8da7e968"
server_key_version: "encryptionkey@0"
I20260806 01:02:42.159062 14255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "e206b800ff4140a494dfd4a520fee1f5"
format_stamp: "Formatted at 2026-08-06 01:02:42 on dist-test-slave-dhph"
server_key: "63cbc4c924de34a143f9f1a94d5f5599"
server_key_iv: "1f1de54bfb00a5cb34fdc79f8da7e968"
server_key_version: "encryptionkey@0"
I20260806 01:02:42.160830 14255 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260806 01:02:42.162142 14270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:42.162474 14255 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260806 01:02:42.162698 14255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "e206b800ff4140a494dfd4a520fee1f5"
format_stamp: "Formatted at 2026-08-06 01:02:42 on dist-test-slave-dhph"
server_key: "63cbc4c924de34a143f9f1a94d5f5599"
server_key_iv: "1f1de54bfb00a5cb34fdc79f8da7e968"
server_key_version: "encryptionkey@0"
I20260806 01:02:42.162808 14255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:42.173092 14255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:42.176156 14255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:42.176530 14255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:42.183876 14255 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:42113
I20260806 01:02:42.183923 14324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:42113 every 8 connection(s)
I20260806 01:02:42.184324 14255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:02:42.185257 14325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:42.187726 14325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e206b800ff4140a494dfd4a520fee1f5: Bootstrap starting.
I20260806 01:02:42.188309 14325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e206b800ff4140a494dfd4a520fee1f5: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:42.188608 14325 log.cc:824] T 00000000000000000000000000000000 P e206b800ff4140a494dfd4a520fee1f5: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:42.189446 14325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e206b800ff4140a494dfd4a520fee1f5: No bootstrap required, opened a new log
I20260806 01:02:42.192202 14325 raft_consensus.cc:348] T 00000000000000000000000000000000 P e206b800ff4140a494dfd4a520fee1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e206b800ff4140a494dfd4a520fee1f5" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 42113 } }
I20260806 01:02:42.192514 14325 raft_consensus.cc:374] T 00000000000000000000000000000000 P e206b800ff4140a494dfd4a520fee1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:42.192546 14325 raft_consensus.cc:729] T 00000000000000000000000000000000 P e206b800ff4140a494dfd4a520fee1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e206b800ff4140a494dfd4a520fee1f5, State: Initialized, Role: FOLLOWER
I20260806 01:02:42.192871 14325 consensus_queue.cc:260] T 00000000000000000000000000000000 P e206b800ff4140a494dfd4a520fee1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e206b800ff4140a494dfd4a520fee1f5" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 42113 } }
I20260806 01:02:42.193028 14325 raft_consensus.cc:388] T 00000000000000000000000000000000 P e206b800ff4140a494dfd4a520fee1f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:02:42.193059 14325 raft_consensus.cc:482] T 00000000000000000000000000000000 P e206b800ff4140a494dfd4a520fee1f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:02:42.193100 14325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e206b800ff4140a494dfd4a520fee1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:42.194151 14325 raft_consensus.cc:504] T 00000000000000000000000000000000 P e206b800ff4140a494dfd4a520fee1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e206b800ff4140a494dfd4a520fee1f5" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 42113 } }
I20260806 01:02:42.194307 14325 leader_election.cc:302] T 00000000000000000000000000000000 P e206b800ff4140a494dfd4a520fee1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e206b800ff4140a494dfd4a520fee1f5; no voters:
I20260806 01:02:42.194458 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 14255
I20260806 01:02:42.194564 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:02:42.194650 14325 leader_election.cc:288] T 00000000000000000000000000000000 P e206b800ff4140a494dfd4a520fee1f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:02:42.194679 14328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e206b800ff4140a494dfd4a520fee1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:42.194815 14328 raft_consensus.cc:686] T 00000000000000000000000000000000 P e206b800ff4140a494dfd4a520fee1f5 [term 1 LEADER]: Becoming Leader. State: Replica: e206b800ff4140a494dfd4a520fee1f5, State: Running, Role: LEADER
I20260806 01:02:42.194865 24520 external_mini_cluster.cc:1428] Setting key 49e1eee30ef41e8b69d3db8367757fb3
I20260806 01:02:42.194978 14325 sys_catalog.cc:564] T 00000000000000000000000000000000 P e206b800ff4140a494dfd4a520fee1f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:42.194962 14328 consensus_queue.cc:237] T 00000000000000000000000000000000 P e206b800ff4140a494dfd4a520fee1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e206b800ff4140a494dfd4a520fee1f5" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 42113 } }
I20260806 01:02:42.195358 14330 sys_catalog.cc:455] T 00000000000000000000000000000000 P e206b800ff4140a494dfd4a520fee1f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e206b800ff4140a494dfd4a520fee1f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e206b800ff4140a494dfd4a520fee1f5" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 42113 } } }
I20260806 01:02:42.195454 14330 sys_catalog.cc:458] T 00000000000000000000000000000000 P e206b800ff4140a494dfd4a520fee1f5 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:42.195914 14336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:02:42.195400 14329 sys_catalog.cc:455] T 00000000000000000000000000000000 P e206b800ff4140a494dfd4a520fee1f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e206b800ff4140a494dfd4a520fee1f5. Latest consensus state: current_term: 1 leader_uuid: "e206b800ff4140a494dfd4a520fee1f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e206b800ff4140a494dfd4a520fee1f5" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 42113 } } }
I20260806 01:02:42.196609 14329 sys_catalog.cc:458] T 00000000000000000000000000000000 P e206b800ff4140a494dfd4a520fee1f5 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:42.196542 14336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:02:42.199066 14336 catalog_manager.cc:1347] Generated new cluster ID: 836c7be7757b41c0a29f35923899cc31
I20260806 01:02:42.199131 14336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 01:02:42 dist-test-slave-dhph krb5kdc[14241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:02:42.205078 14336 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:02:42.205602 14336 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:02:42.207720 14340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:42.195666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53365 (local address 127.23.242.62:42113)
0806 01:02:42.195886 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:42.195896 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:42.195933 (+ 37us) server_negotiation.cc:407] Connection header received
0806 01:02:42.196061 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:42.196068 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:42.196129 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:42.197159 (+ 1030us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:42.197268 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:42.198651 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:42.199682 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:42.200476 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:42.200597 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:42.203104 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:42.203109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:42.203110 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:42.203130 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:42.205906 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:42.206393 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:42.206397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:42.206397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:42.206513 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:42.206779 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:42.206780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:42.206780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:42.207086 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:42.207267 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:42.207377 (+ 110us) server_negotiation.cc:299] Negotiation successful
0806 01:02:42.207484 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":125,"threads_started":1}
I20260806 01:02:42.208796 24520 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 01:02:42.211663 14336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e206b800ff4140a494dfd4a520fee1f5: Generated new TSK 0
I20260806 01:02:42.211862 14336 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 01:02:42.220059 14340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:42.210402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40043 (local address 127.23.242.62:42113)
0806 01:02:42.210449 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:42.210450 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:42.210460 (+ 10us) server_negotiation.cc:407] Connection header received
0806 01:02:42.210507 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:42.210508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:42.210517 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:42.210613 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:42.211343 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:42.212607 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:42.213530 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:42.214179 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:42.214281 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:42.216240 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:42.216242 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:42.216242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:42.216251 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:42.218163 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:42.218743 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:42.218745 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:42.218745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:42.218813 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:42.219064 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:42.219065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:42.219065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:42.219191 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:42.219313 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:42.219851 (+ 538us) server_negotiation.cc:299] Negotiation successful
0806 01:02:42.219926 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 01:02:42.222887 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 14255
W20260806 01:02:42.229681 14239 connection.cc:535] client connection to 127.23.242.62:42113 recv error: Network error: failed to read from TLS socket (remote: 127.23.242.62:42113): Connection reset by peer (error 104)
2026-08-06T01:02:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1517 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 01:02:42 dist-test-slave-dhph krb5kdc[14358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 01:02:42 dist-test-slave-dhph krb5kdc[14358](info): set up 2 sockets
Aug 06 01:02:42 dist-test-slave-dhph krb5kdc[14358](info): commencing operation
krb5kdc: starting...
W20260806 01:02:43.400851 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.155s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 01:02:43 dist-test-slave-dhph krb5kdc[14358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T01:02:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:02:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 01:02:43.571988 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:36973
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:37695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:36973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:43.679023 14371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:43.679284 14371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:43.679315 14371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:43.681804 14371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:43.681892 14371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:43.681910 14371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:43.681923 14371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:43.681937 14371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:43.684175 14371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:37695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:36973
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:36973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:43.684659 14371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:43.685011 14371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:43.691892 14376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:43.691881 14379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:43.691881 14377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:43.692158 14371 server_base.cc:1034] running on GCE node
I20260806 01:02:43.692682 14371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:43.693082 14371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:43.694298 14371 hybrid_clock.cc:648] HybridClock initialized: now 1785978163694314 us; error 69 us; skew 500 ppm
Aug 06 01:02:43 dist-test-slave-dhph krb5kdc[14358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978163, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:43.705745 14371 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 01:02:43.706382 14371 webserver.cc:460] Webserver started at http://127.23.242.62:34629/ using document root <none> and password file <none>
I20260806 01:02:43.706771 14371 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:43.706857 14371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:43.707015 14371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:43.708250 14371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "0f4cd5a44fe34f84bb75577669388b8f"
format_stamp: "Formatted at 2026-08-06 01:02:43 on dist-test-slave-dhph"
server_key: "00a660789dc23129c497171a40518f3f"
server_key_iv: "11a228fa947497b3c02f86edbb19cbce"
server_key_version: "encryptionkey@0"
I20260806 01:02:43.708667 14371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "0f4cd5a44fe34f84bb75577669388b8f"
format_stamp: "Formatted at 2026-08-06 01:02:43 on dist-test-slave-dhph"
server_key: "00a660789dc23129c497171a40518f3f"
server_key_iv: "11a228fa947497b3c02f86edbb19cbce"
server_key_version: "encryptionkey@0"
I20260806 01:02:43.710806 14371 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260806 01:02:43.712149 14386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:43.712577 14371 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:43.712823 14371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "0f4cd5a44fe34f84bb75577669388b8f"
format_stamp: "Formatted at 2026-08-06 01:02:43 on dist-test-slave-dhph"
server_key: "00a660789dc23129c497171a40518f3f"
server_key_iv: "11a228fa947497b3c02f86edbb19cbce"
server_key_version: "encryptionkey@0"
I20260806 01:02:43.712926 14371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:43.738866 14371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:43.741699 14371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:43.741904 14371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:43.747077 14371 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:36973
I20260806 01:02:43.747185 14440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:36973 every 8 connection(s)
I20260806 01:02:43.747594 14371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:02:43.748605 14441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:43.749907 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 14371
I20260806 01:02:43.750021 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:02:43.750254 24520 external_mini_cluster.cc:1428] Setting key 2a8c4a52b7e81b03eebd3d306a7ba515
I20260806 01:02:43.751891 14441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f4cd5a44fe34f84bb75577669388b8f: Bootstrap starting.
I20260806 01:02:43.752722 14441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f4cd5a44fe34f84bb75577669388b8f: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:43.753085 14441 log.cc:824] T 00000000000000000000000000000000 P 0f4cd5a44fe34f84bb75577669388b8f: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:43.753958 14441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f4cd5a44fe34f84bb75577669388b8f: No bootstrap required, opened a new log
I20260806 01:02:43.756688 14441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f4cd5a44fe34f84bb75577669388b8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4cd5a44fe34f84bb75577669388b8f" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 36973 } }
I20260806 01:02:43.756889 14441 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f4cd5a44fe34f84bb75577669388b8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:43.756922 14441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f4cd5a44fe34f84bb75577669388b8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f4cd5a44fe34f84bb75577669388b8f, State: Initialized, Role: FOLLOWER
I20260806 01:02:43.757072 14441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f4cd5a44fe34f84bb75577669388b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4cd5a44fe34f84bb75577669388b8f" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 36973 } }
I20260806 01:02:43.757201 14441 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f4cd5a44fe34f84bb75577669388b8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:02:43.757256 14441 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f4cd5a44fe34f84bb75577669388b8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:02:43.757340 14441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f4cd5a44fe34f84bb75577669388b8f [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:43.758172 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:43.751007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37619 (local address 127.23.242.62:36973)
0806 01:02:43.751241 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:43.751251 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:43.751293 (+ 42us) server_negotiation.cc:407] Connection header received
0806 01:02:43.751898 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:43.751908 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:43.751985 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:43.752163 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:43.752689 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:43.754276 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:43.755258 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:43.756005 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:43.756190 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:43.757341 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:43.757345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:43.757346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:43.757582 (+ 236us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:43.757627 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:43.757643 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:43.757743 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:43.757893 (+ 150us) server_negotiation.cc:299] Negotiation successful
0806 01:02:43.757925 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":116,"threads_started":1}
I20260806 01:02:43.758275 14441 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f4cd5a44fe34f84bb75577669388b8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4cd5a44fe34f84bb75577669388b8f" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 36973 } }
I20260806 01:02:43.758383 14441 leader_election.cc:302] T 00000000000000000000000000000000 P 0f4cd5a44fe34f84bb75577669388b8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f4cd5a44fe34f84bb75577669388b8f; no voters:
I20260806 01:02:43.758630 14441 leader_election.cc:288] T 00000000000000000000000000000000 P 0f4cd5a44fe34f84bb75577669388b8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:02:43.758911 14441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f4cd5a44fe34f84bb75577669388b8f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:43.758924 14447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f4cd5a44fe34f84bb75577669388b8f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:43.759228 14447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f4cd5a44fe34f84bb75577669388b8f [term 1 LEADER]: Becoming Leader. State: Replica: 0f4cd5a44fe34f84bb75577669388b8f, State: Running, Role: LEADER
I20260806 01:02:43.759511 14447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f4cd5a44fe34f84bb75577669388b8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4cd5a44fe34f84bb75577669388b8f" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 36973 } }
I20260806 01:02:43.760098 24520 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 01:02:43.760079 14448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4cd5a44fe34f84bb75577669388b8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f4cd5a44fe34f84bb75577669388b8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4cd5a44fe34f84bb75577669388b8f" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 36973 } } }
I20260806 01:02:43.760165 14448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4cd5a44fe34f84bb75577669388b8f [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:43.760301 14448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4cd5a44fe34f84bb75577669388b8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f4cd5a44fe34f84bb75577669388b8f. Latest consensus state: current_term: 1 leader_uuid: "0f4cd5a44fe34f84bb75577669388b8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4cd5a44fe34f84bb75577669388b8f" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 36973 } } }
I20260806 01:02:43.760490 14448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4cd5a44fe34f84bb75577669388b8f [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:43.760648 14459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:02:43.761904 14459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:02:43.764492 14459 catalog_manager.cc:1347] Generated new cluster ID: 17014fccc2254e998f9cac7233c90114
I20260806 01:02:43.764568 14459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:02:43.766716 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:43.762006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46555 (local address 127.23.242.62:36973)
0806 01:02:43.762050 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:43.762053 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:43.762203 (+ 150us) server_negotiation.cc:407] Connection header received
0806 01:02:43.762228 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:43.762229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:43.762241 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:43.762285 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:43.762479 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:43.763370 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:43.765136 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:43.765451 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:43.765526 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:43.766399 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:43.766400 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:43.766400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:43.766405 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:43.766421 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:43.766425 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:43.766482 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:43.766623 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 01:02:43.766634 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 01:02:43.792562 14459 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:02:43.793154 14459 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:02:43.796998 14459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f4cd5a44fe34f84bb75577669388b8f: Generated new TSK 0
I20260806 01:02:43.797245 14459 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 01:02:43.802999 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 14371
W20260806 01:02:43.809753 14355 connection.cc:535] client connection to 127.23.242.62:36973 recv error: Network error: failed to read from TLS socket (remote: 127.23.242.62:36973): Connection reset by peer (error 104)
2026-08-06T01:02:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1579 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 01:02:43 dist-test-slave-dhph krb5kdc[14472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 01:02:43 dist-test-slave-dhph krb5kdc[14472](info): set up 2 sockets
Aug 06 01:02:43 dist-test-slave-dhph krb5kdc[14472](info): commencing operation
krb5kdc: starting...
W20260806 01:02:45.072629 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.247s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 01:02:45 dist-test-slave-dhph krb5kdc[14472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T01:02:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:02:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 01:02:45.252204 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:38971
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:45845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:38971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:45.356917 14486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:45.357188 14486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:45.357220 14486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:45.359141 14486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:45.359215 14486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:45.359234 14486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:45.359249 14486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:45.359264 14486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:45.362015 14486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:45845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:38971
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:38971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:45.362375 14486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:45.362785 14486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:45.369829 14492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:45.369851 14491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:45.370075 14494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:45.370466 14486 server_base.cc:1034] running on GCE node
I20260806 01:02:45.371121 14486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:45.371537 14486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:45.372766 14486 hybrid_clock.cc:648] HybridClock initialized: now 1785978165372738 us; error 47 us; skew 500 ppm
Aug 06 01:02:45 dist-test-slave-dhph krb5kdc[14472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978165, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:45.376410 14486 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 01:02:45.376998 14486 webserver.cc:460] Webserver started at http://127.23.242.62:45653/ using document root <none> and password file <none>
I20260806 01:02:45.377365 14486 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:45.377447 14486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:45.377602 14486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:45.378957 14486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "b078e895f8a745a18b5af53e669b4271"
format_stamp: "Formatted at 2026-08-06 01:02:45 on dist-test-slave-dhph"
server_key: "2267d558ac376013ade9fb66745c0097"
server_key_iv: "8a165701a0c12bc474612f7f3cc0f3b3"
server_key_version: "encryptionkey@0"
I20260806 01:02:45.379406 14486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "b078e895f8a745a18b5af53e669b4271"
format_stamp: "Formatted at 2026-08-06 01:02:45 on dist-test-slave-dhph"
server_key: "2267d558ac376013ade9fb66745c0097"
server_key_iv: "8a165701a0c12bc474612f7f3cc0f3b3"
server_key_version: "encryptionkey@0"
I20260806 01:02:45.381201 14486 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 01:02:45.382303 14502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:45.382543 14486 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:45.382674 14486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "b078e895f8a745a18b5af53e669b4271"
format_stamp: "Formatted at 2026-08-06 01:02:45 on dist-test-slave-dhph"
server_key: "2267d558ac376013ade9fb66745c0097"
server_key_iv: "8a165701a0c12bc474612f7f3cc0f3b3"
server_key_version: "encryptionkey@0"
I20260806 01:02:45.382771 14486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:45.386677 14486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:45.389914 14486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:45.390093 14486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:45.395059 14486 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:38971
I20260806 01:02:45.395262 14555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:38971 every 8 connection(s)
I20260806 01:02:45.395550 14486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:02:45.396777 14556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:45.399376 14556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b078e895f8a745a18b5af53e669b4271: Bootstrap starting.
I20260806 01:02:45.399757 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 14486
I20260806 01:02:45.399914 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:02:45.400091 14556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b078e895f8a745a18b5af53e669b4271: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:45.400149 24520 external_mini_cluster.cc:1428] Setting key 084dff72861d4a3987c3d14c5e762abd
I20260806 01:02:45.400484 14556 log.cc:824] T 00000000000000000000000000000000 P b078e895f8a745a18b5af53e669b4271: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:45.401407 14556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b078e895f8a745a18b5af53e669b4271: No bootstrap required, opened a new log
I20260806 01:02:45.403826 14560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:45.400851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39757 (local address 127.23.242.62:38971)
0806 01:02:45.401071 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:45.401084 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:45.401113 (+ 29us) server_negotiation.cc:407] Connection header received
0806 01:02:45.402035 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:45.402044 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:45.402075 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:45.402340 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:45.403104 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:45.403108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:45.403108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:45.403113 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:45.403165 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:45.403177 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:45.403305 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:45.403349 (+ 44us) server_negotiation.cc:299] Negotiation successful
0806 01:02:45.403394 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":106,"threads_started":1}
I20260806 01:02:45.404644 14556 raft_consensus.cc:348] T 00000000000000000000000000000000 P b078e895f8a745a18b5af53e669b4271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b078e895f8a745a18b5af53e669b4271" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38971 } }
I20260806 01:02:45.404858 14556 raft_consensus.cc:374] T 00000000000000000000000000000000 P b078e895f8a745a18b5af53e669b4271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:45.404919 14556 raft_consensus.cc:729] T 00000000000000000000000000000000 P b078e895f8a745a18b5af53e669b4271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b078e895f8a745a18b5af53e669b4271, State: Initialized, Role: FOLLOWER
I20260806 01:02:45.404970 24520 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 01:02:45.405074 14556 consensus_queue.cc:260] T 00000000000000000000000000000000 P b078e895f8a745a18b5af53e669b4271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b078e895f8a745a18b5af53e669b4271" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38971 } }
I20260806 01:02:45.405200 14556 raft_consensus.cc:388] T 00000000000000000000000000000000 P b078e895f8a745a18b5af53e669b4271 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:02:45.405254 14556 raft_consensus.cc:482] T 00000000000000000000000000000000 P b078e895f8a745a18b5af53e669b4271 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:02:45.405313 14556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b078e895f8a745a18b5af53e669b4271 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:45.406090 14556 raft_consensus.cc:504] T 00000000000000000000000000000000 P b078e895f8a745a18b5af53e669b4271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b078e895f8a745a18b5af53e669b4271" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38971 } }
I20260806 01:02:45.406343 14556 leader_election.cc:302] T 00000000000000000000000000000000 P b078e895f8a745a18b5af53e669b4271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b078e895f8a745a18b5af53e669b4271; no voters:
I20260806 01:02:45.406752 14556 leader_election.cc:288] T 00000000000000000000000000000000 P b078e895f8a745a18b5af53e669b4271 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:02:45.406865 14565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b078e895f8a745a18b5af53e669b4271 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:45.407109 14565 raft_consensus.cc:686] T 00000000000000000000000000000000 P b078e895f8a745a18b5af53e669b4271 [term 1 LEADER]: Becoming Leader. State: Replica: b078e895f8a745a18b5af53e669b4271, State: Running, Role: LEADER
I20260806 01:02:45.407202 14556 sys_catalog.cc:564] T 00000000000000000000000000000000 P b078e895f8a745a18b5af53e669b4271 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:45.407397 14565 consensus_queue.cc:237] T 00000000000000000000000000000000 P b078e895f8a745a18b5af53e669b4271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b078e895f8a745a18b5af53e669b4271" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38971 } }
I20260806 01:02:45.408532 14565 sys_catalog.cc:455] T 00000000000000000000000000000000 P b078e895f8a745a18b5af53e669b4271 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b078e895f8a745a18b5af53e669b4271. Latest consensus state: current_term: 1 leader_uuid: "b078e895f8a745a18b5af53e669b4271" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b078e895f8a745a18b5af53e669b4271" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38971 } } }
I20260806 01:02:45.408740 14560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:45.406953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40263 (local address 127.23.242.62:38971)
0806 01:02:45.406993 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:45.406995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:45.407283 (+ 288us) server_negotiation.cc:407] Connection header received
0806 01:02:45.407425 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:45.407426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:45.407435 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:45.407479 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:45.408404 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:45.408405 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:45.408405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:45.408411 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:45.408427 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:45.408433 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:45.408471 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:45.408616 (+ 145us) server_negotiation.cc:299] Negotiation successful
0806 01:02:45.408635 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 01:02:45.408846 14565 sys_catalog.cc:458] T 00000000000000000000000000000000 P b078e895f8a745a18b5af53e669b4271 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:45.408736 14568 sys_catalog.cc:455] T 00000000000000000000000000000000 P b078e895f8a745a18b5af53e669b4271 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b078e895f8a745a18b5af53e669b4271" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b078e895f8a745a18b5af53e669b4271" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 38971 } } }
I20260806 01:02:45.409116 14568 sys_catalog.cc:458] T 00000000000000000000000000000000 P b078e895f8a745a18b5af53e669b4271 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:45.409446 14582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:02:45.410291 14582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:02:45.412619 14582 catalog_manager.cc:1347] Generated new cluster ID: 190d9feab0ba47259315c4e4101713cc
I20260806 01:02:45.412738 14582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:02:45.423856 14582 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:02:45.424103 14582 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:02:45.445726 14582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b078e895f8a745a18b5af53e669b4271: Generated new TSK 0
I20260806 01:02:45.446033 14582 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 01:02:45.449968 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 14486
2026-08-06T01:02:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1646 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 01:02:45 dist-test-slave-dhph krb5kdc[14588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 01:02:45 dist-test-slave-dhph krb5kdc[14588](info): set up 2 sockets
Aug 06 01:02:45 dist-test-slave-dhph krb5kdc[14588](info): commencing operation
krb5kdc: starting...
W20260806 01:02:46.751747 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.280s user 0.001s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 01:02:46 dist-test-slave-dhph krb5kdc[14588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T01:02:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:02:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 01:02:46.937804 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:34353
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:34353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:47.043285 14601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:47.043638 14601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:47.043673 14601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:47.046006 14601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:47.046101 14601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:47.046121 14601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:47.046139 14601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:47.046154 14601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:47.048362 14601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:34353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:34353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:47.048628 14601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:47.048866 14601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:47.055631 14609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:47.055660 14607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:47.055711 14601 server_base.cc:1034] running on GCE node
W20260806 01:02:47.055652 14606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:47.056267 14601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:47.056643 14601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:47.057823 14601 hybrid_clock.cc:648] HybridClock initialized: now 1785978167057783 us; error 73 us; skew 500 ppm
Aug 06 01:02:47 dist-test-slave-dhph krb5kdc[14588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978167, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:47.060949 14601 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 01:02:47.061509 14601 webserver.cc:460] Webserver started at http://127.23.242.62:43477/ using document root <none> and password file <none>
I20260806 01:02:47.061803 14601 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:47.061883 14601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:47.062052 14601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:47.063378 14601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "28bc75409e4f4f3ea033e56b8fb674f4"
format_stamp: "Formatted at 2026-08-06 01:02:47 on dist-test-slave-dhph"
server_key: "2bdf08a3383bba3d5007ef5e81852573"
server_key_iv: "95a22352a2a86fedd02571a2ad26aa4b"
server_key_version: "encryptionkey@0"
I20260806 01:02:47.063803 14601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "28bc75409e4f4f3ea033e56b8fb674f4"
format_stamp: "Formatted at 2026-08-06 01:02:47 on dist-test-slave-dhph"
server_key: "2bdf08a3383bba3d5007ef5e81852573"
server_key_iv: "95a22352a2a86fedd02571a2ad26aa4b"
server_key_version: "encryptionkey@0"
I20260806 01:02:47.065572 14601 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260806 01:02:47.067135 14617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:47.067641 14601 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260806 01:02:47.067787 14601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "28bc75409e4f4f3ea033e56b8fb674f4"
format_stamp: "Formatted at 2026-08-06 01:02:47 on dist-test-slave-dhph"
server_key: "2bdf08a3383bba3d5007ef5e81852573"
server_key_iv: "95a22352a2a86fedd02571a2ad26aa4b"
server_key_version: "encryptionkey@0"
I20260806 01:02:47.067862 14601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:47.085172 14601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:47.088258 14601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:47.088399 14601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:47.093703 14601 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:34353
I20260806 01:02:47.093763 14670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:34353 every 8 connection(s)
I20260806 01:02:47.094149 14601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:02:47.095301 14671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:47.096104 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 14601
I20260806 01:02:47.096225 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:02:47.096403 24520 external_mini_cluster.cc:1428] Setting key 01f52289121190177a2dc574abaf0f59
I20260806 01:02:47.098860 14671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28bc75409e4f4f3ea033e56b8fb674f4: Bootstrap starting.
I20260806 01:02:47.099586 14671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28bc75409e4f4f3ea033e56b8fb674f4: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:47.099931 14671 log.cc:824] T 00000000000000000000000000000000 P 28bc75409e4f4f3ea033e56b8fb674f4: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:47.101128 14671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28bc75409e4f4f3ea033e56b8fb674f4: No bootstrap required, opened a new log
I20260806 01:02:47.104238 14671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28bc75409e4f4f3ea033e56b8fb674f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28bc75409e4f4f3ea033e56b8fb674f4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 34353 } }
I20260806 01:02:47.104400 14671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28bc75409e4f4f3ea033e56b8fb674f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:47.104425 14671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28bc75409e4f4f3ea033e56b8fb674f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28bc75409e4f4f3ea033e56b8fb674f4, State: Initialized, Role: FOLLOWER
I20260806 01:02:47.104575 14671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28bc75409e4f4f3ea033e56b8fb674f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28bc75409e4f4f3ea033e56b8fb674f4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 34353 } }
I20260806 01:02:47.104656 14671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28bc75409e4f4f3ea033e56b8fb674f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:02:47.104681 14671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28bc75409e4f4f3ea033e56b8fb674f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:02:47.104753 14671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28bc75409e4f4f3ea033e56b8fb674f4 [term 0 FOLLOWER]: Advancing to term 1
Aug 06 01:02:47 dist-test-slave-dhph krb5kdc[14588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:02:47.105363 14671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28bc75409e4f4f3ea033e56b8fb674f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28bc75409e4f4f3ea033e56b8fb674f4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 34353 } }
I20260806 01:02:47.105441 14671 leader_election.cc:302] T 00000000000000000000000000000000 P 28bc75409e4f4f3ea033e56b8fb674f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28bc75409e4f4f3ea033e56b8fb674f4; no voters:
I20260806 01:02:47.105666 14671 leader_election.cc:288] T 00000000000000000000000000000000 P 28bc75409e4f4f3ea033e56b8fb674f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:02:47.105754 14676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28bc75409e4f4f3ea033e56b8fb674f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:47.105981 14676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28bc75409e4f4f3ea033e56b8fb674f4 [term 1 LEADER]: Becoming Leader. State: Replica: 28bc75409e4f4f3ea033e56b8fb674f4, State: Running, Role: LEADER
I20260806 01:02:47.105984 14671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28bc75409e4f4f3ea033e56b8fb674f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:47.106253 14676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28bc75409e4f4f3ea033e56b8fb674f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28bc75409e4f4f3ea033e56b8fb674f4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 34353 } }
I20260806 01:02:47.108280 14677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28bc75409e4f4f3ea033e56b8fb674f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28bc75409e4f4f3ea033e56b8fb674f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28bc75409e4f4f3ea033e56b8fb674f4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 34353 } } }
I20260806 01:02:47.108495 14677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28bc75409e4f4f3ea033e56b8fb674f4 [sys.catalog]: This master's current role is: LEADER
W20260806 01:02:47.108566 14691 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 28bc75409e4f4f3ea033e56b8fb674f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 01:02:47.108619 14691 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 01:02:47.108775 14678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28bc75409e4f4f3ea033e56b8fb674f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28bc75409e4f4f3ea033e56b8fb674f4. Latest consensus state: current_term: 1 leader_uuid: "28bc75409e4f4f3ea033e56b8fb674f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28bc75409e4f4f3ea033e56b8fb674f4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 34353 } } }
I20260806 01:02:47.108937 14693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:02:47.108958 14678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28bc75409e4f4f3ea033e56b8fb674f4 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:47.109269 14693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:02:47.110168 14674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:47.097055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44877 (local address 127.23.242.62:34353)
0806 01:02:47.097301 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:47.097311 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:47.097345 (+ 34us) server_negotiation.cc:407] Connection header received
0806 01:02:47.097973 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:47.097983 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:47.098050 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:47.098215 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:47.099389 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:47.101002 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:47.102087 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:47.102893 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:47.103027 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:47.106024 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:47.106029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:47.106029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:47.106043 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:47.108381 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:47.108863 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:47.108867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:47.108868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:47.108982 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:47.109220 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:47.109221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:47.109222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:47.109535 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:47.109726 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:47.109852 (+ 126us) server_negotiation.cc:299] Negotiation successful
0806 01:02:47.110001 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":138,"threads_started":1}
I20260806 01:02:47.111253 14693 catalog_manager.cc:1347] Generated new cluster ID: f2e3a18ac33049e5aae8ebd44e3c9513
I20260806 01:02:47.111312 14693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:02:47.111647 24520 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 01:02:47.122360 14674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:47.115076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48849 (local address 127.23.242.62:34353)
0806 01:02:47.115141 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:47.115143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:47.115455 (+ 312us) server_negotiation.cc:407] Connection header received
0806 01:02:47.115532 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:47.115533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:47.115544 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:47.115630 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:47.116266 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:47.116848 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:47.117645 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:47.117908 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:47.117996 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:47.119417 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:47.119418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:47.119419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:47.119427 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:47.121014 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:47.121459 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:47.121460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:47.121461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:47.121526 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:47.121760 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:47.121761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:47.121761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:47.121891 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:47.122004 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:47.122149 (+ 145us) server_negotiation.cc:299] Negotiation successful
0806 01:02:47.122240 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 01:02:47.129549 14693 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:02:47.130249 14693 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:02:47.134123 14693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28bc75409e4f4f3ea033e56b8fb674f4: Generated new TSK 0
I20260806 01:02:47.134344 14693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 01:02:47.142617 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 14601
2026-08-06T01:02:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1693 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 01:02:47 dist-test-slave-dhph krb5kdc[14702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 01:02:47 dist-test-slave-dhph krb5kdc[14702](info): set up 2 sockets
Aug 06 01:02:47 dist-test-slave-dhph krb5kdc[14702](info): commencing operation
krb5kdc: starting...
W20260806 01:02:48.399219 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.234s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 01:02:48 dist-test-slave-dhph krb5kdc[14702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T01:02:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:02:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 01:02:48.567040 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:44587
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:44427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:44587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:48.673475 14716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:48.673751 14716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:48.673782 14716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:48.675832 14716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:48.675925 14716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:48.675945 14716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:48.675961 14716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:48.675976 14716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:48.679886 14716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:44427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:44587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:44587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:48.680289 14716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:48.680601 14716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:48.688611 14722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:48.688611 14724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:48.688616 14721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:48.688645 14716 server_base.cc:1034] running on GCE node
I20260806 01:02:48.689427 14716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:48.689798 14716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:48.691025 14716 hybrid_clock.cc:648] HybridClock initialized: now 1785978168690981 us; error 69 us; skew 500 ppm
Aug 06 01:02:48 dist-test-slave-dhph krb5kdc[14702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978168, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:48.694747 14716 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 01:02:48.695300 14716 webserver.cc:460] Webserver started at http://127.23.242.62:35053/ using document root <none> and password file <none>
I20260806 01:02:48.695600 14716 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:48.695683 14716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:48.695858 14716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:48.697387 14716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "0815ccc832f84d55a65c60e3cff3f0b7"
format_stamp: "Formatted at 2026-08-06 01:02:48 on dist-test-slave-dhph"
server_key: "6b5192fb38b2178f3d10799a3f1dfc1e"
server_key_iv: "84ea739b6861345f49c39dc4839b9b87"
server_key_version: "encryptionkey@0"
I20260806 01:02:48.697851 14716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "0815ccc832f84d55a65c60e3cff3f0b7"
format_stamp: "Formatted at 2026-08-06 01:02:48 on dist-test-slave-dhph"
server_key: "6b5192fb38b2178f3d10799a3f1dfc1e"
server_key_iv: "84ea739b6861345f49c39dc4839b9b87"
server_key_version: "encryptionkey@0"
I20260806 01:02:48.699671 14716 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260806 01:02:48.701081 14732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:48.701541 14716 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:48.701673 14716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "0815ccc832f84d55a65c60e3cff3f0b7"
format_stamp: "Formatted at 2026-08-06 01:02:48 on dist-test-slave-dhph"
server_key: "6b5192fb38b2178f3d10799a3f1dfc1e"
server_key_iv: "84ea739b6861345f49c39dc4839b9b87"
server_key_version: "encryptionkey@0"
I20260806 01:02:48.701773 14716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:48.731793 14716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:48.734773 14716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:48.734964 14716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:48.741362 14716 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:44587
I20260806 01:02:48.741467 14785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:44587 every 8 connection(s)
I20260806 01:02:48.741887 14716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:02:48.743484 14786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:48.744944 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 14716
I20260806 01:02:48.745220 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:02:48.745481 24520 external_mini_cluster.cc:1428] Setting key 417bb8d112983da5173a53b01537d634
I20260806 01:02:48.748070 14786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0815ccc832f84d55a65c60e3cff3f0b7: Bootstrap starting.
I20260806 01:02:48.748901 14786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0815ccc832f84d55a65c60e3cff3f0b7: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:48.749274 14786 log.cc:824] T 00000000000000000000000000000000 P 0815ccc832f84d55a65c60e3cff3f0b7: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:48.750147 14786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0815ccc832f84d55a65c60e3cff3f0b7: No bootstrap required, opened a new log
I20260806 01:02:48.752887 14789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:48.746238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52697 (local address 127.23.242.62:44587)
0806 01:02:48.746453 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:48.746461 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:48.746491 (+ 30us) server_negotiation.cc:407] Connection header received
0806 01:02:48.747031 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:48.747039 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:48.747091 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:48.747225 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:48.747641 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:48.748610 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:48.749906 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:48.750522 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:48.750727 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:48.751703 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:48.751708 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:48.751708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:48.751927 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:48.751969 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:48.751985 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:48.752276 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:48.752445 (+ 169us) server_negotiation.cc:299] Negotiation successful
0806 01:02:48.752533 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":109,"threads_started":1}
I20260806 01:02:48.753425 14786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0815ccc832f84d55a65c60e3cff3f0b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0815ccc832f84d55a65c60e3cff3f0b7" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44587 } }
I20260806 01:02:48.753592 14786 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0815ccc832f84d55a65c60e3cff3f0b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:48.753619 14786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0815ccc832f84d55a65c60e3cff3f0b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0815ccc832f84d55a65c60e3cff3f0b7, State: Initialized, Role: FOLLOWER
I20260806 01:02:48.753777 14786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0815ccc832f84d55a65c60e3cff3f0b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0815ccc832f84d55a65c60e3cff3f0b7" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44587 } }
I20260806 01:02:48.753893 14786 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0815ccc832f84d55a65c60e3cff3f0b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:02:48.753962 14786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0815ccc832f84d55a65c60e3cff3f0b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:02:48.754021 14786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0815ccc832f84d55a65c60e3cff3f0b7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:48.754136 24520 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 01:02:48.754874 14786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0815ccc832f84d55a65c60e3cff3f0b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0815ccc832f84d55a65c60e3cff3f0b7" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44587 } }
I20260806 01:02:48.755016 14786 leader_election.cc:302] T 00000000000000000000000000000000 P 0815ccc832f84d55a65c60e3cff3f0b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0815ccc832f84d55a65c60e3cff3f0b7; no voters:
I20260806 01:02:48.755271 14786 leader_election.cc:288] T 00000000000000000000000000000000 P 0815ccc832f84d55a65c60e3cff3f0b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:02:48.755317 14794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0815ccc832f84d55a65c60e3cff3f0b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:48.755645 14794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0815ccc832f84d55a65c60e3cff3f0b7 [term 1 LEADER]: Becoming Leader. State: Replica: 0815ccc832f84d55a65c60e3cff3f0b7, State: Running, Role: LEADER
I20260806 01:02:48.755689 14786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0815ccc832f84d55a65c60e3cff3f0b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:48.755832 14794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0815ccc832f84d55a65c60e3cff3f0b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0815ccc832f84d55a65c60e3cff3f0b7" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44587 } }
I20260806 01:02:48.756320 14798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0815ccc832f84d55a65c60e3cff3f0b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0815ccc832f84d55a65c60e3cff3f0b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0815ccc832f84d55a65c60e3cff3f0b7" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44587 } } }
I20260806 01:02:48.756439 14798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0815ccc832f84d55a65c60e3cff3f0b7 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:48.756345 14799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0815ccc832f84d55a65c60e3cff3f0b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0815ccc832f84d55a65c60e3cff3f0b7. Latest consensus state: current_term: 1 leader_uuid: "0815ccc832f84d55a65c60e3cff3f0b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0815ccc832f84d55a65c60e3cff3f0b7" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 44587 } } }
I20260806 01:02:48.756538 14799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0815ccc832f84d55a65c60e3cff3f0b7 [sys.catalog]: This master's current role is: LEADER
W20260806 01:02:48.758562 14814 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0815ccc832f84d55a65c60e3cff3f0b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 01:02:48.758716 14814 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 01:02:48.758844 14804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:02:48.759080 14804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:02:48.761287 14789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:48.755816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40351 (local address 127.23.242.62:44587)
0806 01:02:48.755869 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:48.755871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:48.756221 (+ 350us) server_negotiation.cc:407] Connection header received
0806 01:02:48.756368 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:48.756369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:48.756382 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:48.756419 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:48.756987 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:48.757822 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:48.758945 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:48.759252 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:48.759302 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:48.760948 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:48.760950 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:48.760950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:48.760957 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:48.760977 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:48.760982 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:48.761018 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:48.761152 (+ 134us) server_negotiation.cc:299] Negotiation successful
0806 01:02:48.761188 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 01:02:48.761629 14804 catalog_manager.cc:1347] Generated new cluster ID: 48a3c705f3e848a79cddb38c787981cf
I20260806 01:02:48.761744 14804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:02:48.780237 14804 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:02:48.781543 14804 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:02:48.795732 14804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0815ccc832f84d55a65c60e3cff3f0b7: Generated new TSK 0
I20260806 01:02:48.795991 14804 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 01:02:48.804006 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 14716
2026-08-06T01:02:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1660 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 01:02:48 dist-test-slave-dhph krb5kdc[14819](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 01:02:48 dist-test-slave-dhph krb5kdc[14819](info): set up 2 sockets
Aug 06 01:02:48 dist-test-slave-dhph krb5kdc[14819](info): commencing operation
krb5kdc: starting...
W20260806 01:02:50.080756 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.255s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 01:02:50 dist-test-slave-dhph krb5kdc[14819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T01:02:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:02:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 01:02:50.244634 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:40033
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:34631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:40033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:50.337256 14832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:50.337476 14832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:50.337499 14832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:50.338701 14832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:50.338752 14832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:50.338768 14832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:50.338780 14832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:50.338794 14832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:50.340273 14832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:34631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:40033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:40033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:50.340530 14832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:50.340766 14832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:50.346395 14840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:50.346395 14838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:50.346514 14837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:50.346698 14832 server_base.cc:1034] running on GCE node
I20260806 01:02:50.347272 14832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:50.347625 14832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:50.348836 14832 hybrid_clock.cc:648] HybridClock initialized: now 1785978170348802 us; error 56 us; skew 500 ppm
Aug 06 01:02:50 dist-test-slave-dhph krb5kdc[14819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978170, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:50.352205 14832 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 01:02:50.352880 14832 webserver.cc:460] Webserver started at http://127.23.242.62:43937/ using document root <none> and password file <none>
I20260806 01:02:50.353212 14832 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:50.353294 14832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:50.353451 14832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:50.354569 14832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "0f0881d5b368487d94b245575d220adf"
format_stamp: "Formatted at 2026-08-06 01:02:50 on dist-test-slave-dhph"
server_key: "474414e0436ffe82d5647e59427b96c2"
server_key_iv: "8c75664c33e27b84e4a1bcceff7ce105"
server_key_version: "encryptionkey@0"
I20260806 01:02:50.355015 14832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "0f0881d5b368487d94b245575d220adf"
format_stamp: "Formatted at 2026-08-06 01:02:50 on dist-test-slave-dhph"
server_key: "474414e0436ffe82d5647e59427b96c2"
server_key_iv: "8c75664c33e27b84e4a1bcceff7ce105"
server_key_version: "encryptionkey@0"
I20260806 01:02:50.356513 14832 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 01:02:50.357476 14847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:50.357697 14832 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 01:02:50.357836 14832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "0f0881d5b368487d94b245575d220adf"
format_stamp: "Formatted at 2026-08-06 01:02:50 on dist-test-slave-dhph"
server_key: "474414e0436ffe82d5647e59427b96c2"
server_key_iv: "8c75664c33e27b84e4a1bcceff7ce105"
server_key_version: "encryptionkey@0"
I20260806 01:02:50.357919 14832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:50.362195 14832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:50.364812 14832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:50.365096 14832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:50.369637 14832 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:40033
I20260806 01:02:50.369773 14901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:40033 every 8 connection(s)
I20260806 01:02:50.370087 14832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:02:50.371309 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 14832
I20260806 01:02:50.371325 14902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:50.371481 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:02:50.371719 24520 external_mini_cluster.cc:1428] Setting key 6d6e3eca6945d4a8ff4e54736851bce8
I20260806 01:02:50.374505 14902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f0881d5b368487d94b245575d220adf: Bootstrap starting.
I20260806 01:02:50.375293 14902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f0881d5b368487d94b245575d220adf: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:50.375651 14902 log.cc:824] T 00000000000000000000000000000000 P 0f0881d5b368487d94b245575d220adf: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:50.376390 14902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f0881d5b368487d94b245575d220adf: No bootstrap required, opened a new log
I20260806 01:02:50.378248 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:50.372496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59763 (local address 127.23.242.62:40033)
0806 01:02:50.372723 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:50.372734 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:50.374947 (+ 2213us) server_negotiation.cc:407] Connection header received
0806 01:02:50.376326 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:50.376337 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:50.376370 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:50.376634 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:50.377648 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:50.377651 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:50.377651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:50.377656 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:50.377700 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:50.377711 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:50.377803 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:50.377990 (+ 187us) server_negotiation.cc:299] Negotiation successful
0806 01:02:50.378048 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":102,"threads_started":1}
I20260806 01:02:50.379055 14902 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f0881d5b368487d94b245575d220adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f0881d5b368487d94b245575d220adf" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 40033 } }
I20260806 01:02:50.379266 14902 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f0881d5b368487d94b245575d220adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 01:02:50.379315 14902 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f0881d5b368487d94b245575d220adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0881d5b368487d94b245575d220adf, State: Initialized, Role: FOLLOWER
I20260806 01:02:50.379442 14902 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f0881d5b368487d94b245575d220adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f0881d5b368487d94b245575d220adf" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 40033 } }
I20260806 01:02:50.379550 14902 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f0881d5b368487d94b245575d220adf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 01:02:50.379604 14902 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f0881d5b368487d94b245575d220adf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 01:02:50.379647 14902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f0881d5b368487d94b245575d220adf [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:50.379846 24520 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 01:02:50.380481 14902 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f0881d5b368487d94b245575d220adf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f0881d5b368487d94b245575d220adf" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 40033 } }
I20260806 01:02:50.380612 14902 leader_election.cc:302] T 00000000000000000000000000000000 P 0f0881d5b368487d94b245575d220adf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f0881d5b368487d94b245575d220adf; no voters:
I20260806 01:02:50.380810 14902 leader_election.cc:288] T 00000000000000000000000000000000 P 0f0881d5b368487d94b245575d220adf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 01:02:50.380962 14907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f0881d5b368487d94b245575d220adf [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:50.381135 14902 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f0881d5b368487d94b245575d220adf [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:50.381295 14907 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f0881d5b368487d94b245575d220adf [term 1 LEADER]: Becoming Leader. State: Replica: 0f0881d5b368487d94b245575d220adf, State: Running, Role: LEADER
I20260806 01:02:50.381527 14907 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f0881d5b368487d94b245575d220adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f0881d5b368487d94b245575d220adf" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 40033 } }
I20260806 01:02:50.382022 14912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f0881d5b368487d94b245575d220adf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f0881d5b368487d94b245575d220adf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f0881d5b368487d94b245575d220adf" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 40033 } } }
I20260806 01:02:50.382126 14912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f0881d5b368487d94b245575d220adf [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:50.382383 14917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:02:50.382407 14908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f0881d5b368487d94b245575d220adf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f0881d5b368487d94b245575d220adf. Latest consensus state: current_term: 1 leader_uuid: "0f0881d5b368487d94b245575d220adf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f0881d5b368487d94b245575d220adf" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 40033 } } }
I20260806 01:02:50.382586 14908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f0881d5b368487d94b245575d220adf [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:50.383435 14917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:02:50.384166 14905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:50.381896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37531 (local address 127.23.242.62:40033)
0806 01:02:50.381945 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:50.381947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:50.382309 (+ 362us) server_negotiation.cc:407] Connection header received
0806 01:02:50.382465 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:50.382466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:50.382474 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:50.382558 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:50.383785 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:50.383788 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:50.383789 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 01:02:50.383797 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:50.383832 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 01:02:50.383840 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:50.383886 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:50.384046 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 01:02:50.384067 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 01:02:50.387954 14917 catalog_manager.cc:1347] Generated new cluster ID: e2d8e9a795fd4273b760f28bea8bbe01
I20260806 01:02:50.388051 14917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:02:50.397327 14917 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:02:50.397537 14917 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:02:50.412401 14917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f0881d5b368487d94b245575d220adf: Generated new TSK 0
I20260806 01:02:50.412672 14917 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 01:02:50.415436 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 14832
2026-08-06T01:02:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1610 ms)
[----------] 18 tests from AuthTokenIssuingTest (28346 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 01:02:50 dist-test-slave-dhph krb5kdc[14934](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 01:02:50 dist-test-slave-dhph krb5kdc[14934](info): set up 2 sockets
Aug 06 01:02:50 dist-test-slave-dhph krb5kdc[14934](info): commencing operation
krb5kdc: starting...
W20260806 01:02:51.644352 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.208s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 01:02:51 dist-test-slave-dhph krb5kdc[14934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T01:02:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:02:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 01:02:51.817670 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:43239
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:44497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:43239,127.23.242.61:43061,127.23.242.60:43583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:51.917799 14947 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 01:02:51.918120 14947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:51.918164 14947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:51.918215 14947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:51.921161 14947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:51.921352 14947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:51.921387 14947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:51.921411 14947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:51.921435 14947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:51.924686 14947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:44497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:43239,127.23.242.61:43061,127.23.242.60:43583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:43239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:51.925051 14947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:51.925328 14947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:51.930840 14952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:51.930893 14953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:51.930932 14956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:51.931346 14947 server_base.cc:1034] running on GCE node
I20260806 01:02:51.931679 14947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:51.932201 14947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:51.933564 14947 hybrid_clock.cc:648] HybridClock initialized: now 1785978171933344 us; error 184 us; skew 500 ppm
Aug 06 01:02:51 dist-test-slave-dhph krb5kdc[14934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978171, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:51.937218 14947 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 01:02:51.937860 14947 webserver.cc:460] Webserver started at http://127.23.242.62:44033/ using document root <none> and password file <none>
I20260806 01:02:51.938153 14947 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:51.938232 14947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:51.938401 14947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:51.939908 14947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "4e2bf4fa46a74ea3973e2bf0f00be780"
format_stamp: "Formatted at 2026-08-06 01:02:51 on dist-test-slave-dhph"
server_key: "e4269fde8db96d3910e4b245770ecda2"
server_key_iv: "09d148a62d88bbe2ec785a4b7b91e089"
server_key_version: "encryptionkey@0"
I20260806 01:02:51.940436 14947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "4e2bf4fa46a74ea3973e2bf0f00be780"
format_stamp: "Formatted at 2026-08-06 01:02:51 on dist-test-slave-dhph"
server_key: "e4269fde8db96d3910e4b245770ecda2"
server_key_iv: "09d148a62d88bbe2ec785a4b7b91e089"
server_key_version: "encryptionkey@0"
I20260806 01:02:51.942059 14947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 01:02:51.943331 14964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:51.943717 14947 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 01:02:51.943859 14947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "4e2bf4fa46a74ea3973e2bf0f00be780"
format_stamp: "Formatted at 2026-08-06 01:02:51 on dist-test-slave-dhph"
server_key: "e4269fde8db96d3910e4b245770ecda2"
server_key_iv: "09d148a62d88bbe2ec785a4b7b91e089"
server_key_version: "encryptionkey@0"
I20260806 01:02:51.943956 14947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:51.959122 14947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:51.963076 14947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:51.963335 14947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:51.969867 14947 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:43239
I20260806 01:02:51.970327 14947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:02:51.969964 15018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:43239 every 8 connection(s)
I20260806 01:02:51.971751 15020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:51.973449 15020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43239 } has no permanent_uuid. Determining permanent_uuid...
I20260806 01:02:51.975525 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 14947
I20260806 01:02:51.975731 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:02:51.976123 24520 external_mini_cluster.cc:1428] Setting key ce0cb5f4a79347133ace986f5d24e788
Aug 06 01:02:51 dist-test-slave-dhph krb5kdc[14934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978171, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.242.61@KRBTEST.COM; defaulting to no policy
I20260806 01:02:51.989162 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:51.974373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52306 (local address 127.23.242.62:43239)
0806 01:02:51.974747 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:51.974765 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:51.974849 (+ 84us) server_negotiation.cc:407] Connection header received
0806 01:02:51.976057 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:51.976059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:51.976123 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:51.976241 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:51.977155 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:51.978530 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:51.979733 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:51.980265 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:51.980398 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:51.984287 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:51.984297 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:51.984298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:51.984307 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:51.987231 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:51.987775 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:51.987779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:51.987780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:51.987896 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:51.988169 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:51.988170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:51.988170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:51.988440 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:51.988549 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:51.988821 (+ 272us) server_negotiation.cc:299] Negotiation successful
0806 01:02:51.988946 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":108,"threads_started":1}
I20260806 01:02:51.989491 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:51.974274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43239 (local address 127.0.0.1:52306)
0806 01:02:51.974544 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:51.974745 (+ 201us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:51.975491 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:51.976249 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:51.976252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:51.976585 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:51.976920 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:51.976928 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:51.978753 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:51.978754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:51.979614 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:51.979617 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:51.980536 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:51.980538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:51.980637 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:51.981543 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:02:51.981554 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:51.984145 (+ 2591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:51.987366 (+ 3221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:51.987375 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:51.987377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:51.987619 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:51.987990 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:51.987991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:51.987991 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:51.988090 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:51.988552 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:51.988559 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:51.988628 (+ 69us) client_negotiation.cc:769] Sending connection context
0806 01:02:51.988724 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 01:02:51.989197 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":186,"thread_start_us":162,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.61@KRBTEST.COM" created.
I20260806 01:02:51.990684 15020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.61" port: 43061 } has no permanent_uuid. Determining permanent_uuid...
I20260806 01:02:51.991253 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:51.991082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:43061 (local address 127.0.0.1:38458)
0806 01:02:51.991173 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:51.991228 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.61:43061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 01:02:51.991403 14968 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.242.61:43061: connect: Connection refused (error 111)
I20260806 01:02:51.992379 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:51.992184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:43061 (local address 127.0.0.1:38462)
0806 01:02:51.992319 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:51.992354 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.61:43061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260806 01:02:51.992563 15020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.242.61:43061: Network error: Client connection negotiation failed: client connection to 127.23.242.61:43061: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.61.keytab.
Entry for principal kudu/127.23.242.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.61.keytab.
I20260806 01:02:52.020576 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.61:43061
--webserver_interface=127.23.242.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.61.keytab
--principal=kudu/127.23.242.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:44497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:43239,127.23.242.61:43061,127.23.242.60:43583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 01:02:52.041939 15020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.242.61" port: 43061 } attempt: 1
I20260806 01:02:52.042531 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.042384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:43061 (local address 127.0.0.1:38472)
0806 01:02:52.042458 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:52.042499 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.61:43061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 01:02:52.043051 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.042934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:43061 (local address 127.0.0.1:38478)
0806 01:02:52.043001 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:52.043030 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.61:43061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 01:02:52.043179 15020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.242.61:43061: Network error: Client connection negotiation failed: client connection to 127.23.242.61:43061: connect: Connection refused (error 111)
I20260806 01:02:52.111363 15020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.242.61" port: 43061 } attempt: 2
I20260806 01:02:52.111917 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.111755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:43061 (local address 127.0.0.1:38484)
0806 01:02:52.111841 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:52.111885 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.61:43061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 01:02:52.112352 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.112245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:43061 (local address 127.0.0.1:38486)
0806 01:02:52.112298 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:52.112326 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.61:43061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 01:02:52.112458 15020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.242.61:43061: Network error: Client connection negotiation failed: client connection to 127.23.242.61:43061: connect: Connection refused (error 111)
W20260806 01:02:52.118940 15028 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 01:02:52.119309 15028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:52.119359 15028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:52.119410 15028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:52.120879 15028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:52.120971 15028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:52.121004 15028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:52.121052 15028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:52.121090 15028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:52.123203 15028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:44497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:43239,127.23.242.61:43061,127.23.242.60:43583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.61.keytab
--principal=kudu/127.23.242.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.61:43061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.242.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:52.123564 15028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:52.124012 15028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:52.130427 15037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:52.130376 15035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:52.130425 15028 server_base.cc:1034] running on GCE node
W20260806 01:02:52.130846 15034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:52.131441 15028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:52.132100 15028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:52.133347 15028 hybrid_clock.cc:648] HybridClock initialized: now 1785978172133325 us; error 46 us; skew 500 ppm
Aug 06 01:02:52 dist-test-slave-dhph krb5kdc[14934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978172, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:52.136844 15028 init.cc:377] Logged in from keytab as kudu/127.23.242.61@KRBTEST.COM (short username kudu)
I20260806 01:02:52.137380 15028 webserver.cc:460] Webserver started at http://127.23.242.61:41279/ using document root <none> and password file <none>
I20260806 01:02:52.137658 15028 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:52.137741 15028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:52.137899 15028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:52.139139 15028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-1/data/instance:
uuid: "d2f638de582c4ee087d0a755b1c9caac"
format_stamp: "Formatted at 2026-08-06 01:02:52 on dist-test-slave-dhph"
server_key: "3bf5c3d3b5d39420d4c884b858bba17f"
server_key_iv: "d704fd0b831af2dbadc3cf503a98b320"
server_key_version: "encryptionkey@0"
I20260806 01:02:52.139518 15028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-1/wal/instance:
uuid: "d2f638de582c4ee087d0a755b1c9caac"
format_stamp: "Formatted at 2026-08-06 01:02:52 on dist-test-slave-dhph"
server_key: "3bf5c3d3b5d39420d4c884b858bba17f"
server_key_iv: "d704fd0b831af2dbadc3cf503a98b320"
server_key_version: "encryptionkey@0"
I20260806 01:02:52.141068 15028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 01:02:52.142272 15045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:52.142729 15028 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:52.142863 15028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-1/wal
uuid: "d2f638de582c4ee087d0a755b1c9caac"
format_stamp: "Formatted at 2026-08-06 01:02:52 on dist-test-slave-dhph"
server_key: "3bf5c3d3b5d39420d4c884b858bba17f"
server_key_iv: "d704fd0b831af2dbadc3cf503a98b320"
server_key_version: "encryptionkey@0"
I20260806 01:02:52.142987 15028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:52.174419 15028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:52.177424 15028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:52.177641 15028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:52.182663 15028 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.61:43061
I20260806 01:02:52.182757 15098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.61:43061 every 8 connection(s)
I20260806 01:02:52.183133 15028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-1/data/info.pb
I20260806 01:02:52.184221 15099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:52.185840 15099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43239 } has no permanent_uuid. Determining permanent_uuid...
I20260806 01:02:52.188946 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 15028
I20260806 01:02:52.189074 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-1/wal/instance
I20260806 01:02:52.189298 24520 external_mini_cluster.cc:1428] Setting key 11dfe9f99ff9be0afee2ae9272918b55
Aug 06 01:02:52 dist-test-slave-dhph krb5kdc[14934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978172, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.61@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:02:52.198450 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.186701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52314 (local address 127.23.242.62:43239)
0806 01:02:52.186761 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:52.186763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:52.186995 (+ 232us) server_negotiation.cc:407] Connection header received
0806 01:02:52.187444 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:52.187445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:52.187460 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:52.187536 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:52.188876 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.189587 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.191233 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.191627 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.191727 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:52.194843 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:52.194844 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:52.194844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:52.194852 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:52.196539 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.197197 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.197198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.197198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.197265 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.197597 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.197598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.197598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.197766 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:52.197826 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:52.198287 (+ 461us) server_negotiation.cc:299] Negotiation successful
0806 01:02:52.198373 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260806 01:02:52.198745 15101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.186524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43239 (local address 127.0.0.1:52314)
0806 01:02:52.186773 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:52.186817 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:52.187297 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:52.187754 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:52.187761 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:52.188194 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:52.188560 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:52.188567 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.189707 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.189708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:52.191096 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:52.191101 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.191804 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.191806 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:52.191924 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:52.192540 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:02:52.192549 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:52.194702 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:52.196648 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:52.196659 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:52.196662 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:52.197056 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:52.197350 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:52.197351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:52.197352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:52.197509 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:52.197857 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:52.197865 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:52.198075 (+ 210us) client_negotiation.cc:769] Sending connection context
0806 01:02:52.198227 (+ 152us) client_negotiation.cc:240] Negotiation successful
0806 01:02:52.198439 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":154,"threads_started":1}
WARNING: no policy specified for kudu/127.23.242.60@KRBTEST.COM; defaulting to no policy
I20260806 01:02:52.199419 15099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.61" port: 43061 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.60@KRBTEST.COM" created.
I20260806 01:02:52.203615 15020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.242.61" port: 43061 } attempt: 3
Aug 06 01:02:52 dist-test-slave-dhph krb5kdc[14934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978172, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.61@KRBTEST.COM for kudu/127.23.242.61@KRBTEST.COM
Aug 06 01:02:52 dist-test-slave-dhph krb5kdc[14934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978171, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for kudu/127.23.242.61@KRBTEST.COM
I20260806 01:02:52.209589 15101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.200054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:43061 (local address 127.0.0.1:38494)
0806 01:02:52.200290 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:52.200366 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:52.200427 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:52.200737 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:52.200738 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:52.200904 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:52.200963 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:52.200965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.201945 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.201947 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:52.202571 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:52.202573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.203222 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.203223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:52.203303 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:52.203923 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:02:52.203929 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:52.206072 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:52.208436 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:52.208438 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:52.208438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:52.208611 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:52.208855 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:52.208855 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:52.208856 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:52.208893 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:52.209222 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:52.209223 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:52.209265 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 01:02:52.209328 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 01:02:52.209443 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
I20260806 01:02:52.209618 15103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.200319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38494 (local address 127.23.242.61:43061)
0806 01:02:52.200581 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:52.200586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:52.200603 (+ 17us) server_negotiation.cc:407] Connection header received
0806 01:02:52.200622 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:52.200624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:52.200637 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:52.200738 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:52.201060 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.201829 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.202734 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.203137 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.203228 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:52.206185 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:52.206198 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:52.206198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:52.206205 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:52.208352 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.208724 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.208728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.208729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.208793 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.208957 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.208958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.208958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.209140 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:52.209234 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:52.209375 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 01:02:52.209520 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":237,"thread_start_us":141,"threads_started":1}
I20260806 01:02:52.210570 15099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 } has no permanent_uuid. Determining permanent_uuid...
I20260806 01:02:52.211030 15101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.210899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:43583 (local address 127.0.0.1:49122)
0806 01:02:52.210951 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:52.210999 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:43583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260806 01:02:52.211252 15047 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.242.60:43583: connect: Connection refused (error 111)
I20260806 01:02:52.211766 15101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.211661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:43583 (local address 127.0.0.1:49128)
0806 01:02:52.211715 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:52.211741 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:43583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260806 01:02:52.211979 15099 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.242.60:43583: Network error: Client connection negotiation failed: client connection to 127.23.242.60:43583: connect: Connection refused (error 111)
I20260806 01:02:52.213750 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.203971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:43061 (local address 127.0.0.1:38502)
0806 01:02:52.204035 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:52.204049 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:52.204104 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:52.204372 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:52.204374 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:52.204569 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:52.204639 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:52.204642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.205607 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.205608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:52.206487 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:52.206490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.207027 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.207028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:52.207120 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:52.207622 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:02:52.207628 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:52.209315 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:52.212328 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:52.212329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:52.212329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:52.212521 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:52.212819 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:52.212820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:52.212820 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:52.212902 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:52.213397 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:52.213398 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:52.213450 (+ 52us) client_negotiation.cc:769] Sending connection context
0806 01:02:52.213537 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 01:02:52.213638 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260806 01:02:52.213831 15104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.204088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38502 (local address 127.23.242.61:43061)
0806 01:02:52.204232 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:52.204234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:52.204242 (+ 8us) server_negotiation.cc:407] Connection header received
0806 01:02:52.204262 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:52.204263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:52.204273 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:52.204343 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:52.204737 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.205496 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.206618 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.206936 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.207010 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:52.209421 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:52.209423 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:52.209423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:52.209430 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:52.212220 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.212611 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.212612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.212613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.212679 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.212986 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.212987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.212987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.213261 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:52.213349 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:52.213597 (+ 248us) server_negotiation.cc:299] Negotiation successful
0806 01:02:52.213685 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260806 01:02:52.214151 15020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 } has no permanent_uuid. Determining permanent_uuid...
I20260806 01:02:52.214509 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.214394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:43583 (local address 127.0.0.1:49140)
0806 01:02:52.214447 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:52.214477 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:43583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260806 01:02:52.215595 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.215443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:43583 (local address 127.0.0.1:49152)
0806 01:02:52.215530 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:52.215571 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:43583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 01:02:52.216034 15020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.242.60:43583: Network error: Client connection negotiation failed: client connection to 127.23.242.60:43583: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.60.keytab.
Entry for principal kudu/127.23.242.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.60.keytab.
I20260806 01:02:52.230505 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.60:43583
--webserver_interface=127.23.242.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.60.keytab
--principal=kudu/127.23.242.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:44497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:43239,127.23.242.61:43061,127.23.242.60:43583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 01:02:52.247290 15020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 } attempt: 1
I20260806 01:02:52.248270 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.248100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:43583 (local address 127.0.0.1:49154)
0806 01:02:52.248186 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:52.248230 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:43583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 01:02:52.248711 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.248555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:43583 (local address 127.0.0.1:49156)
0806 01:02:52.248648 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:52.248678 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:43583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260806 01:02:52.248977 15020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.242.60:43583: Network error: Client connection negotiation failed: client connection to 127.23.242.60:43583: connect: Connection refused (error 111)
I20260806 01:02:52.261264 15099 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 } attempt: 1
I20260806 01:02:52.261788 15101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.261652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:43583 (local address 127.0.0.1:49172)
0806 01:02:52.261718 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:52.261760 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:43583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260806 01:02:52.262339 15101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.262176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:43583 (local address 127.0.0.1:49174)
0806 01:02:52.262249 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:52.262317 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:43583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 01:02:52.262486 15099 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.242.60:43583: Network error: Client connection negotiation failed: client connection to 127.23.242.60:43583: connect: Connection refused (error 111)
I20260806 01:02:52.324185 15020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 } attempt: 2
I20260806 01:02:52.325117 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.324932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:43583 (local address 127.0.0.1:49180)
0806 01:02:52.325020 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:52.325075 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:43583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 01:02:52.326002 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.325839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:43583 (local address 127.0.0.1:49184)
0806 01:02:52.325922 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:52.325967 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:43583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 01:02:52.326498 15020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.242.60:43583: Network error: Client connection negotiation failed: client connection to 127.23.242.60:43583: connect: Connection refused (error 111)
I20260806 01:02:52.330711 15099 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 } attempt: 2
I20260806 01:02:52.331274 15101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.331088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:43583 (local address 127.0.0.1:49196)
0806 01:02:52.331188 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:52.331231 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:43583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260806 01:02:52.331646 15107 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 01:02:52.332289 15107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:52.332316 15107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:52.332345 15107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 01:02:52.332360 15101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.332084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:43583 (local address 127.0.0.1:49200)
0806 01:02:52.332195 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:52.332294 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:43583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260806 01:02:52.332609 15099 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.242.60:43583: Network error: Client connection negotiation failed: client connection to 127.23.242.60:43583: connect: Connection refused (error 111)
W20260806 01:02:52.334337 15107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:52.334446 15107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:52.334481 15107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:52.334525 15107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:52.334555 15107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:52.336767 15107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:44497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:43239,127.23.242.61:43061,127.23.242.60:43583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.60.keytab
--principal=kudu/127.23.242.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.60:43583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.242.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:52.337124 15107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:52.337404 15107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:52.344539 15115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:52.344561 15113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:52.344599 15107 server_base.cc:1034] running on GCE node
W20260806 01:02:52.344573 15117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:52.345119 15107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:52.345461 15107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:52.346829 15107 hybrid_clock.cc:648] HybridClock initialized: now 1785978172346789 us; error 80 us; skew 500 ppm
Aug 06 01:02:52 dist-test-slave-dhph krb5kdc[14934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978172, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:52.350741 15107 init.cc:377] Logged in from keytab as kudu/127.23.242.60@KRBTEST.COM (short username kudu)
I20260806 01:02:52.351486 15107 webserver.cc:460] Webserver started at http://127.23.242.60:42321/ using document root <none> and password file <none>
I20260806 01:02:52.351795 15107 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:52.352159 15107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:52.352380 15107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:52.354540 15107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-2/data/instance:
uuid: "c3d1e99054d148ef95b47b72c23a3ef4"
format_stamp: "Formatted at 2026-08-06 01:02:52 on dist-test-slave-dhph"
server_key: "071d3647478a00d565c953363803112b"
server_key_iv: "cab3edddf7f6902cfde6fe72badf4904"
server_key_version: "encryptionkey@0"
I20260806 01:02:52.355038 15107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-2/wal/instance:
uuid: "c3d1e99054d148ef95b47b72c23a3ef4"
format_stamp: "Formatted at 2026-08-06 01:02:52 on dist-test-slave-dhph"
server_key: "071d3647478a00d565c953363803112b"
server_key_iv: "cab3edddf7f6902cfde6fe72badf4904"
server_key_version: "encryptionkey@0"
I20260806 01:02:52.356670 15107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260806 01:02:52.357781 15125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:52.358136 15107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:52.358310 15107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-2/wal
uuid: "c3d1e99054d148ef95b47b72c23a3ef4"
format_stamp: "Formatted at 2026-08-06 01:02:52 on dist-test-slave-dhph"
server_key: "071d3647478a00d565c953363803112b"
server_key_iv: "cab3edddf7f6902cfde6fe72badf4904"
server_key_version: "encryptionkey@0"
I20260806 01:02:52.358407 15107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:52.368423 15107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:52.371109 15107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:52.371317 15107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:52.376436 15107 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.60:43583
I20260806 01:02:52.376484 15180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.60:43583 every 8 connection(s)
I20260806 01:02:52.376890 15107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-2/data/info.pb
I20260806 01:02:52.377983 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 15107
I20260806 01:02:52.378118 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785977869987768-24520-0/minicluster-data/master-2/wal/instance
I20260806 01:02:52.378127 15181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:52.378332 24520 external_mini_cluster.cc:1428] Setting key 2d371c6d6da02aff4fe3791c12293b01
I20260806 01:02:52.381183 15181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43239 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 01:02:52 dist-test-slave-dhph krb5kdc[14934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.60@KRBTEST.COM
Aug 06 01:02:52 dist-test-slave-dhph krb5kdc[14934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978172, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.60@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:02:52.393551 15184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.379143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.23.242.60:43583)
0806 01:02:52.379632 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:52.379644 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:52.379682 (+ 38us) server_negotiation.cc:407] Connection header received
0806 01:02:52.380177 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:52.380185 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:52.380236 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:52.380366 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:52.381558 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.383414 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.384822 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.385505 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.385609 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:52.388956 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:52.388967 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:52.388967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:52.388975 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:52.391229 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.391770 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.391777 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.391777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.391879 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.392131 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.392132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.392132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.392422 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:52.392615 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:52.392722 (+ 107us) server_negotiation.cc:299] Negotiation successful
0806 01:02:52.393375 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":394,"server-negotiator.queue_time_us":245,"thread_start_us":84,"threads_started":1}
I20260806 01:02:52.393551 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.382093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52322 (local address 127.23.242.62:43239)
0806 01:02:52.382159 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:52.382161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:52.382901 (+ 740us) server_negotiation.cc:407] Connection header received
0806 01:02:52.383101 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:52.383108 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:52.383121 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:52.383191 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:52.384430 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.385145 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.386316 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.386501 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.386619 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:52.389570 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:52.389572 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:52.389572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:52.389609 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:52.391476 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.391886 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.391887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.391888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.391949 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.392196 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.392197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.392197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.392336 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:52.392412 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:52.392622 (+ 210us) server_negotiation.cc:299] Negotiation successful
0806 01:02:52.393176 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 01:02:52.393599 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.381914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:43239 (local address 127.0.0.1:52322)
0806 01:02:52.382547 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:52.382713 (+ 166us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:52.382968 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:52.383241 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:52.383245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:52.384151 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:52.384287 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:52.384290 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.385292 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.385293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:52.386175 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:52.386179 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.386667 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.386668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:52.386737 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:52.387297 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:02:52.387306 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:52.389315 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:52.391562 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:52.391570 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:52.391572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:52.391787 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:52.392018 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:52.392019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:52.392020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:52.392121 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:52.392405 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:52.392412 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:52.392485 (+ 73us) client_negotiation.cc:769] Sending connection context
0806 01:02:52.392599 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 01:02:52.393096 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":505,"threads_started":1}
I20260806 01:02:52.394790 15181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.61" port: 43061 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 01:02:52 dist-test-slave-dhph krb5kdc[14934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
Aug 06 01:02:52 dist-test-slave-dhph krb5kdc[14934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978172, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.60@KRBTEST.COM for kudu/127.23.242.61@KRBTEST.COM
I20260806 01:02:52.404008 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.394771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52336 (local address 127.23.242.62:43239)
0806 01:02:52.394819 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:52.394821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:52.394834 (+ 13us) server_negotiation.cc:407] Connection header received
0806 01:02:52.394923 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:52.394924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:52.394936 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:52.394994 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:52.395646 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.396177 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.396988 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.397210 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.397299 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:52.400195 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:52.400197 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:52.400198 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:52.400206 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:52.402693 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.403111 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.403112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.403113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.403178 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.403394 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.403395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.403395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.403540 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:52.403729 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:52.403768 (+ 39us) server_negotiation.cc:299] Negotiation successful
0806 01:02:52.403873 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 01:02:52.405337 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.395106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:43061 (local address 127.0.0.1:38504)
0806 01:02:52.395150 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:52.395159 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:52.395191 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:52.395404 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:52.395405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:52.395564 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:52.395603 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:52.395604 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.396591 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.396593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:52.397228 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:52.397231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.397737 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.397738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:52.397814 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:52.398438 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:02:52.398443 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:52.400943 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:52.403673 (+ 2730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:52.403675 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:52.403676 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:52.403899 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:52.404367 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:52.404368 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:52.404369 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:52.404448 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:52.404972 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:52.404973 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:52.405033 (+ 60us) client_negotiation.cc:769] Sending connection context
0806 01:02:52.405111 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 01:02:52.405236 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260806 01:02:52.405356 15104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.395211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38504 (local address 127.23.242.61:43061)
0806 01:02:52.395278 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:52.395280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:52.395287 (+ 7us) server_negotiation.cc:407] Connection header received
0806 01:02:52.395311 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:52.395312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:52.395326 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:52.395400 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:52.395682 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.396401 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.397348 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.397626 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.397735 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:52.401108 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:52.401110 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:52.401111 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:52.401125 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:52.403540 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.403999 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.404000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.404000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.404093 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.404634 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.404635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.404636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.404844 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:52.404917 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:52.405140 (+ 223us) server_negotiation.cc:299] Negotiation successful
0806 01:02:52.405237 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 01:02:52.406067 15181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 01:02:52 dist-test-slave-dhph krb5kdc[14934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.61@KRBTEST.COM
Aug 06 01:02:52 dist-test-slave-dhph krb5kdc[14934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978172, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.60@KRBTEST.COM for kudu/127.23.242.60@KRBTEST.COM
I20260806 01:02:52.418533 15103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.405023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38518 (local address 127.23.242.61:43061)
0806 01:02:52.405089 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:52.405091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:52.405098 (+ 7us) server_negotiation.cc:407] Connection header received
0806 01:02:52.405124 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:52.405125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:52.405135 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:52.405200 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:52.405759 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.406318 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.407436 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.407836 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.407932 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:52.412853 (+ 4921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:52.412855 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:52.412856 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:52.412866 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:52.416476 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.417078 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.417080 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.417081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.417155 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.417512 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.417513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.417513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.417753 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:52.417949 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:52.418288 (+ 339us) server_negotiation.cc:299] Negotiation successful
0806 01:02:52.418395 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 01:02:52.419008 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.406393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:43583 (local address 127.0.0.1:49220)
0806 01:02:52.406446 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:52.406458 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:52.406496 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:52.406641 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:52.406642 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:52.406780 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:52.406813 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:52.406815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.407719 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.407720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:52.408409 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:52.408412 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.408844 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.408845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:52.408912 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:52.409322 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:02:52.409328 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:52.413746 (+ 4418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:52.417256 (+ 3510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:52.417257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:52.417258 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:52.417514 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:52.417833 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:52.417834 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:52.417835 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:52.417943 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:52.418535 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:52.418536 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:52.418622 (+ 86us) client_negotiation.cc:769] Sending connection context
0806 01:02:52.418757 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 01:02:52.418899 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260806 01:02:52.419202 15184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.406434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49220 (local address 127.23.242.60:43583)
0806 01:02:52.406474 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:52.406476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:52.406486 (+ 10us) server_negotiation.cc:407] Connection header received
0806 01:02:52.406545 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:52.406545 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:52.406555 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:52.406638 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:52.406898 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.407617 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.408492 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.408741 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.408806 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:52.413859 (+ 5053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:52.413861 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:52.413861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:52.413870 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:52.417074 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.417602 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.417603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.417603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.417666 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.418268 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.418269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.418269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.418415 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:52.418489 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:52.418802 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 01:02:52.418994 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":19}
I20260806 01:02:52.421442 24520 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 01:02:52.421934 15181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3d1e99054d148ef95b47b72c23a3ef4: Bootstrap starting.
I20260806 01:02:52.423012 15181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3d1e99054d148ef95b47b72c23a3ef4: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:52.423360 15181 log.cc:824] T 00000000000000000000000000000000 P c3d1e99054d148ef95b47b72c23a3ef4: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:52.424018 15099 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 } attempt: 3
I20260806 01:02:52.424294 15181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3d1e99054d148ef95b47b72c23a3ef4: No bootstrap required, opened a new log
I20260806 01:02:52.425720 15020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 } attempt: 3
I20260806 01:02:52.427197 15181 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3d1e99054d148ef95b47b72c23a3ef4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43239 } } peers { permanent_uuid: "d2f638de582c4ee087d0a755b1c9caac" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 43061 } } peers { permanent_uuid: "c3d1e99054d148ef95b47b72c23a3ef4" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 } }
I20260806 01:02:52.427376 15181 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3d1e99054d148ef95b47b72c23a3ef4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3d1e99054d148ef95b47b72c23a3ef4, State: Initialized, Role: FOLLOWER
I20260806 01:02:52.427474 15181 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3d1e99054d148ef95b47b72c23a3ef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43239 } } peers { permanent_uuid: "d2f638de582c4ee087d0a755b1c9caac" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 43061 } } peers { permanent_uuid: "c3d1e99054d148ef95b47b72c23a3ef4" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 } }
I20260806 01:02:52.425438 14992 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52336
I20260806 01:02:52.427829 15188 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3d1e99054d148ef95b47b72c23a3ef4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43239 } } peers { permanent_uuid: "d2f638de582c4ee087d0a755b1c9caac" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 43061 } } peers { permanent_uuid: "c3d1e99054d148ef95b47b72c23a3ef4" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 } } }
I20260806 01:02:52.427888 15181 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3d1e99054d148ef95b47b72c23a3ef4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:52.427968 15188 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3d1e99054d148ef95b47b72c23a3ef4 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 01:02:52.430228 15199 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c3d1e99054d148ef95b47b72c23a3ef4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 01:02:52.431051 15199 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 06 01:02:52 dist-test-slave-dhph krb5kdc[14934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978172, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.61@KRBTEST.COM for kudu/127.23.242.60@KRBTEST.COM
I20260806 01:02:52.431902 15107 master_runner.cc:186] Error getting master registration for 127.23.242.62:43239: OK, instance_id { permanent_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" instance_seqno: 1785978171968413 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 01:02:52.435436 15107 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 06 01:02:52 dist-test-slave-dhph krb5kdc[14934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978171, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for kudu/127.23.242.60@KRBTEST.COM
I20260806 01:02:52.438506 15184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.424510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49226 (local address 127.23.242.60:43583)
0806 01:02:52.424589 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:52.424591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:52.424603 (+ 12us) server_negotiation.cc:407] Connection header received
0806 01:02:52.424630 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:52.424631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:52.424642 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:52.424716 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:52.425139 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.426060 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.427519 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.427828 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.428317 (+ 489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:52.433094 (+ 4777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:52.433095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:52.433095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:52.433104 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:52.436077 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.436704 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.436706 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.436706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.436780 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.437041 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.437042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.437042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.437270 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:52.437353 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:52.438284 (+ 931us) server_negotiation.cc:299] Negotiation successful
0806 01:02:52.438388 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 01:02:52.438340 15101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.424356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:43583 (local address 127.0.0.1:49226)
0806 01:02:52.424481 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:52.424499 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:52.424569 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:52.424738 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:52.424739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:52.424964 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:52.425034 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:52.425037 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.426495 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.426497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:52.427400 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:52.427404 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.427958 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.427960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:52.428028 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:52.430518 (+ 2490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:02:52.430525 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:52.432728 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:52.436223 (+ 3495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:52.436224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:52.436224 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:52.436619 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:52.436871 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:52.436873 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:52.436873 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:52.436958 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:52.437719 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:52.437720 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:52.437785 (+ 65us) client_negotiation.cc:769] Sending connection context
0806 01:02:52.437886 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 01:02:52.438017 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260806 01:02:52.441921 15099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f638de582c4ee087d0a755b1c9caac: Bootstrap starting.
I20260806 01:02:52.442993 15099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2f638de582c4ee087d0a755b1c9caac: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:52.443356 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.425975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:43583 (local address 127.0.0.1:49240)
0806 01:02:52.426027 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:52.426039 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:52.426077 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:52.430516 (+ 4439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:52.430518 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:52.430760 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:52.430818 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:52.430828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.432657 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.432659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:52.433623 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:52.433628 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.434204 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.434206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:52.434291 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:52.434819 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:02:52.434825 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:52.436905 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:52.440934 (+ 4029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:52.440936 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:52.440937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:52.441145 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:52.441989 (+ 844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:52.441990 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:52.441990 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:52.442124 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:52.442939 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:52.442940 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:52.443014 (+ 74us) client_negotiation.cc:769] Sending connection context
0806 01:02:52.443110 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 01:02:52.443240 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260806 01:02:52.443373 15099 log.cc:824] T 00000000000000000000000000000000 P d2f638de582c4ee087d0a755b1c9caac: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:52.444201 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.426213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49240 (local address 127.23.242.60:43583)
0806 01:02:52.428724 (+ 2511us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:52.428727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:52.428762 (+ 35us) server_negotiation.cc:407] Connection header received
0806 01:02:52.428785 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:52.428786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:52.428797 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:52.430956 (+ 2159us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:52.431014 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.432540 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.433745 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.434105 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.434472 (+ 367us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:52.437015 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:52.437017 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:52.437018 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:52.437026 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:52.440846 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.441470 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.441471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.441472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.441876 (+ 404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.442204 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.442205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.442206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.442441 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:52.442946 (+ 505us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:52.443340 (+ 394us) server_negotiation.cc:299] Negotiation successful
0806 01:02:52.443963 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2484,"thread_start_us":794,"threads_started":1}
I20260806 01:02:52.444720 15099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f638de582c4ee087d0a755b1c9caac: No bootstrap required, opened a new log
I20260806 01:02:52.446427 15020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780: Bootstrap starting.
I20260806 01:02:52.447576 15020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:52.447450 15099 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2f638de582c4ee087d0a755b1c9caac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43239 } } peers { permanent_uuid: "d2f638de582c4ee087d0a755b1c9caac" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 43061 } } peers { permanent_uuid: "c3d1e99054d148ef95b47b72c23a3ef4" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 } }
I20260806 01:02:52.447649 15099 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2f638de582c4ee087d0a755b1c9caac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2f638de582c4ee087d0a755b1c9caac, State: Initialized, Role: FOLLOWER
I20260806 01:02:52.447759 15099 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2f638de582c4ee087d0a755b1c9caac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43239 } } peers { permanent_uuid: "d2f638de582c4ee087d0a755b1c9caac" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 43061 } } peers { permanent_uuid: "c3d1e99054d148ef95b47b72c23a3ef4" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 } }
I20260806 01:02:52.447956 15020 log.cc:824] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:52.448141 15099 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2f638de582c4ee087d0a755b1c9caac [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:52.448974 15020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780: No bootstrap required, opened a new log
W20260806 01:02:52.450443 15215 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d2f638de582c4ee087d0a755b1c9caac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 01:02:52.450542 15215 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 01:02:52.450860 15028 master_runner.cc:186] Error getting master registration for 127.23.242.62:43239: OK, instance_id { permanent_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" instance_seqno: 1785978171968413 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 01:02:52.452720 15028 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 01:02:52.453415 15020 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43239 } } peers { permanent_uuid: "d2f638de582c4ee087d0a755b1c9caac" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 43061 } } peers { permanent_uuid: "c3d1e99054d148ef95b47b72c23a3ef4" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 } }
I20260806 01:02:52.453584 15020 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e2bf4fa46a74ea3973e2bf0f00be780, State: Initialized, Role: FOLLOWER
I20260806 01:02:52.453691 15020 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43239 } } peers { permanent_uuid: "d2f638de582c4ee087d0a755b1c9caac" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 43061 } } peers { permanent_uuid: "c3d1e99054d148ef95b47b72c23a3ef4" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 } }
I20260806 01:02:52.454137 15020 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:52.454219 14992 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 01:02:52.454314 14992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:52.455363 14992 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43239 } } peers { permanent_uuid: "d2f638de582c4ee087d0a755b1c9caac" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 43061 } } peers { permanent_uuid: "c3d1e99054d148ef95b47b72c23a3ef4" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 } }
I20260806 01:02:52.455659 14992 leader_election.cc:288] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780 [CANDIDATE]: Term 1 election: Requested vote from peers d2f638de582c4ee087d0a755b1c9caac (127.23.242.61:43061), c3d1e99054d148ef95b47b72c23a3ef4 (127.23.242.60:43583)
W20260806 01:02:52.456378 15228 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260806 01:02:52.456364 15072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d2f638de582c4ee087d0a755b1c9caac"
I20260806 01:02:52.456516 15072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2f638de582c4ee087d0a755b1c9caac [term 0 FOLLOWER]: Advancing to term 1
W20260806 01:02:52.456610 15228 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 01:02:52.457576 15072 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d2f638de582c4ee087d0a755b1c9caac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e2bf4fa46a74ea3973e2bf0f00be780 in term 1.
I20260806 01:02:52.457641 15153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c3d1e99054d148ef95b47b72c23a3ef4"
I20260806 01:02:52.457803 15153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3d1e99054d148ef95b47b72c23a3ef4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:52.457847 14968 leader_election.cc:302] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e2bf4fa46a74ea3973e2bf0f00be780, d2f638de582c4ee087d0a755b1c9caac; no voters:
I20260806 01:02:52.458885 15153 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c3d1e99054d148ef95b47b72c23a3ef4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e2bf4fa46a74ea3973e2bf0f00be780 in term 1.
I20260806 01:02:52.459712 15236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43239 } } peers { permanent_uuid: "d2f638de582c4ee087d0a755b1c9caac" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 43061 } } peers { permanent_uuid: "c3d1e99054d148ef95b47b72c23a3ef4" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 } } }
I20260806 01:02:52.459935 15236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 01:02:52.460062 15236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:52.460335 15236 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780 [term 1 LEADER]: Becoming Leader. State: Replica: 4e2bf4fa46a74ea3973e2bf0f00be780, State: Running, Role: LEADER
I20260806 01:02:52.460671 15236 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43239 } } peers { permanent_uuid: "d2f638de582c4ee087d0a755b1c9caac" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 43061 } } peers { permanent_uuid: "c3d1e99054d148ef95b47b72c23a3ef4" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 } }
I20260806 01:02:52.461588 15236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e2bf4fa46a74ea3973e2bf0f00be780. Latest consensus state: current_term: 1 leader_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43239 } } peers { permanent_uuid: "d2f638de582c4ee087d0a755b1c9caac" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 43061 } } peers { permanent_uuid: "c3d1e99054d148ef95b47b72c23a3ef4" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 } } }
I20260806 01:02:52.461778 15236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:52.462126 14947 master_runner.cc:263] No leader master found from master 4e2bf4fa46a74ea3973e2bf0f00be780
I20260806 01:02:52.462183 14947 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 01:02:52.462697 15202 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f638de582c4ee087d0a755b1c9caac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43239 } } peers { permanent_uuid: "d2f638de582c4ee087d0a755b1c9caac" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 43061 } } peers { permanent_uuid: "c3d1e99054d148ef95b47b72c23a3ef4" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 } } }
I20260806 01:02:52.462808 15202 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f638de582c4ee087d0a755b1c9caac [sys.catalog]: This master's current role is: FOLLOWER
W20260806 01:02:52.464216 14970 tablet.cc:2369] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 01:02:52.474861 15240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:02:52.475140 15240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:02:52.481844 15153 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c3d1e99054d148ef95b47b72c23a3ef4 [term 1 FOLLOWER]: Refusing update from remote peer 4e2bf4fa46a74ea3973e2bf0f00be780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:02:52.482333 15072 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d2f638de582c4ee087d0a755b1c9caac [term 1 FOLLOWER]: Refusing update from remote peer 4e2bf4fa46a74ea3973e2bf0f00be780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:02:52.482728 15217 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2f638de582c4ee087d0a755b1c9caac" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 43061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:02:52.483228 15217 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3d1e99054d148ef95b47b72c23a3ef4" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:02:52.484931 15202 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f638de582c4ee087d0a755b1c9caac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e2bf4fa46a74ea3973e2bf0f00be780. Latest consensus state: current_term: 1 leader_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43239 } } peers { permanent_uuid: "d2f638de582c4ee087d0a755b1c9caac" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 43061 } } peers { permanent_uuid: "c3d1e99054d148ef95b47b72c23a3ef4" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 } } }
I20260806 01:02:52.485076 15202 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f638de582c4ee087d0a755b1c9caac [sys.catalog]: This master's current role is: FOLLOWER
I20260806 01:02:52.487231 15188 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3d1e99054d148ef95b47b72c23a3ef4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e2bf4fa46a74ea3973e2bf0f00be780. Latest consensus state: current_term: 1 leader_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43239 } } peers { permanent_uuid: "d2f638de582c4ee087d0a755b1c9caac" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 43061 } } peers { permanent_uuid: "c3d1e99054d148ef95b47b72c23a3ef4" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 } } }
I20260806 01:02:52.487367 15188 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3d1e99054d148ef95b47b72c23a3ef4 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 01:02:52.489485 15103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.459533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38530 (local address 127.23.242.61:43061)
0806 01:02:52.459580 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:52.459582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:52.463293 (+ 3711us) server_negotiation.cc:407] Connection header received
0806 01:02:52.463391 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:52.463392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:52.463404 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:52.463438 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:52.465005 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.465834 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.477181 (+ 11347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.477590 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.477642 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:52.479315 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:52.479316 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:52.479317 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:52.479324 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:52.481158 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.488017 (+ 6859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.488019 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.488020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.488100 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.488342 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.488343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.488344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.488517 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:52.488622 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:52.489253 (+ 631us) server_negotiation.cc:299] Negotiation successful
0806 01:02:52.489362 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 01:02:52.491093 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.457778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52352 (local address 127.23.242.62:43239)
0806 01:02:52.458086 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:52.458088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:52.461995 (+ 3907us) server_negotiation.cc:407] Connection header received
0806 01:02:52.462472 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:52.462474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:52.462487 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:52.462548 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:52.463544 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.464394 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.465692 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.466060 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.466120 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:52.474029 (+ 7909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:52.474032 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:52.474032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:52.474043 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:52.475965 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.478352 (+ 2387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.478354 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.478355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.478439 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.490276 (+ 11837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.490278 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.490279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.490474 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:52.490609 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:52.490868 (+ 259us) server_negotiation.cc:299] Negotiation successful
0806 01:02:52.490979 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281}
I20260806 01:02:52.492494 15217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43239 } } peers { permanent_uuid: "d2f638de582c4ee087d0a755b1c9caac" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 43061 } } peers { permanent_uuid: "c3d1e99054d148ef95b47b72c23a3ef4" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 } } }
I20260806 01:02:52.492705 15217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:52.493309 15202 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f638de582c4ee087d0a755b1c9caac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43239 } } peers { permanent_uuid: "d2f638de582c4ee087d0a755b1c9caac" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 43061 } } peers { permanent_uuid: "c3d1e99054d148ef95b47b72c23a3ef4" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 } } }
I20260806 01:02:52.493422 15202 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f638de582c4ee087d0a755b1c9caac [sys.catalog]: This master's current role is: FOLLOWER
I20260806 01:02:52.493496 15242 mvcc.cc:205] Tried to move back new op lower bound from 7315366594477555712 to 7315366594401648640. Current Snapshot: MvccSnapshot[applied={T|T < 7315366594477555712}]
I20260806 01:02:52.495035 15236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43239 } } peers { permanent_uuid: "d2f638de582c4ee087d0a755b1c9caac" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 43061 } } peers { permanent_uuid: "c3d1e99054d148ef95b47b72c23a3ef4" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 } } }
I20260806 01:02:52.495160 15236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:52.495898 15188 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3d1e99054d148ef95b47b72c23a3ef4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e2bf4fa46a74ea3973e2bf0f00be780" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 43239 } } peers { permanent_uuid: "d2f638de582c4ee087d0a755b1c9caac" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 43061 } } peers { permanent_uuid: "c3d1e99054d148ef95b47b72c23a3ef4" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 43583 } } }
I20260806 01:02:52.496025 15188 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3d1e99054d148ef95b47b72c23a3ef4 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 01:02:52.496361 15244 mvcc.cc:205] Tried to move back new op lower bound from 7315366594477555712 to 7315366594401648640. Current Snapshot: MvccSnapshot[applied={T|T < 7315366594477555712}]
I20260806 01:02:52.497313 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:52.459082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49254 (local address 127.23.242.60:43583)
0806 01:02:52.462265 (+ 3183us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:52.462267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:52.469227 (+ 6960us) server_negotiation.cc:407] Connection header received
0806 01:02:52.469264 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:52.469265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:52.469282 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:52.469344 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:52.470566 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.471475 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.472914 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:52.473310 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:52.473389 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:52.477755 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:52.477757 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:52.477758 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:52.477767 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:52.493057 (+ 15290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.493939 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.493940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.493940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.494004 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:52.494202 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:52.494202 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:52.494203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:52.494347 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:52.494430 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:52.496998 (+ 2568us) server_negotiation.cc:299] Negotiation successful
0806 01:02:52.497159 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3157}
I20260806 01:02:52.499192 15240 catalog_manager.cc:1347] Generated new cluster ID: 52ad35c6d9f9410a97b7443d6780072b
I20260806 01:02:52.499279 15240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:02:52.550911 15240 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:02:52.551726 15240 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:02:52.563513 15240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e2bf4fa46a74ea3973e2bf0f00be780: Generated new TSK 0
I20260806 01:02:52.563797 15240 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 01:02:52.594022 15103 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:02:52.592344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38532 (local address 127.23.242.61:43061)
0806 01:02:52.592406 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:52.592408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:52.593175 (+ 767us) server_negotiation.cc:407] Connection header received
0806 01:02:52.593297 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:52.593298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:52.593312 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:52.593346 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:52.593926 (+ 580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38532: BlockingRecv error: recv got EOF from 127.0.0.1:38532 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260806 01:02:52.596860 15187 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:02:52.592861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49262 (local address 127.23.242.60:43583)
0806 01:02:52.592909 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:52.592911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:52.594535 (+ 1624us) server_negotiation.cc:407] Connection header received
0806 01:02:52.594694 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:52.594696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:52.594708 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:52.594741 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:52.596756 (+ 2015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49262: BlockingRecv error: recv got EOF from 127.0.0.1:49262 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260806 01:02:52.598753 15023 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:02:52.597505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52360 (local address 127.23.242.62:43239)
0806 01:02:52.597560 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:52.597562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:52.597857 (+ 295us) server_negotiation.cc:407] Connection header received
0806 01:02:52.597979 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:52.597981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:52.597995 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:52.598030 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:52.598624 (+ 594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52360: BlockingRecv error: recv got EOF from 127.0.0.1:52360 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260806 01:02:52.602546 15103 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:02:52.601109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38542 (local address 127.23.242.61:43061)
0806 01:02:52.601164 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:52.601167 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:52.601633 (+ 466us) server_negotiation.cc:407] Connection header received
0806 01:02:52.601820 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:52.601821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:52.601849 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:52.601897 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:52.602478 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38542: BlockingRecv error: recv got EOF from 127.0.0.1:38542 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260806 01:02:52.606328 15103 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:02:52.605278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38554 (local address 127.23.242.61:43061)
0806 01:02:52.605330 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:52.605332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:52.605642 (+ 310us) server_negotiation.cc:407] Connection header received
0806 01:02:52.605750 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:52.605752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:52.605764 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:52.605797 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:52.606240 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38554: BlockingRecv error: recv got EOF from 127.0.0.1:38554 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260806 01:02:52.613611 15103 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:02:52.611929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38568 (local address 127.23.242.61:43061)
0806 01:02:52.612047 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:52.612051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:52.612609 (+ 558us) server_negotiation.cc:407] Connection header received
0806 01:02:52.612738 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:52.612740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:52.612757 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:52.612792 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:52.613373 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38568: BlockingRecv error: recv got EOF from 127.0.0.1:38568 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260806 01:02:52.630811 15103 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:02:52.629473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38572 (local address 127.23.242.61:43061)
0806 01:02:52.629532 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:52.629534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:52.629833 (+ 299us) server_negotiation.cc:407] Connection header received
0806 01:02:52.629985 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:52.629987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:52.630002 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:52.630037 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:52.630519 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38572: BlockingRecv error: recv got EOF from 127.0.0.1:38572 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260806 01:02:52.645582 15103 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:02:52.644429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38588 (local address 127.23.242.61:43061)
0806 01:02:52.644487 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:52.644489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:52.644837 (+ 348us) server_negotiation.cc:407] Connection header received
0806 01:02:52.644960 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:52.644961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:52.644974 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:52.645010 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:52.645511 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38588: BlockingRecv error: recv got EOF from 127.0.0.1:38588 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260806 01:02:52.666085 15103 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:02:52.663384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38596 (local address 127.23.242.61:43061)
0806 01:02:52.663433 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:52.663435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:52.664217 (+ 782us) server_negotiation.cc:407] Connection header received
0806 01:02:52.664357 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:52.664359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:52.664372 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:52.664406 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:52.665647 (+ 1241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38596: BlockingRecv error: recv got EOF from 127.0.0.1:38596 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260806 01:02:52.703598 15103 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:02:52.700689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38600 (local address 127.23.242.61:43061)
0806 01:02:52.700751 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:52.700754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:52.701186 (+ 432us) server_negotiation.cc:407] Connection header received
0806 01:02:52.701336 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:52.701338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:52.701353 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:52.701395 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:52.703521 (+ 2126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38600: BlockingRecv error: recv got EOF from 127.0.0.1:38600 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260806 01:02:52.771716 15103 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:02:52.770519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38602 (local address 127.23.242.61:43061)
0806 01:02:52.770668 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:52.770670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:52.770857 (+ 187us) server_negotiation.cc:407] Connection header received
0806 01:02:52.770889 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:52.770890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:52.770902 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:52.770963 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:52.771635 (+ 672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38602: BlockingRecv error: recv got EOF from 127.0.0.1:38602 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
W20260806 01:02:52.902742 15103 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:02:52.901617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38604 (local address 127.23.242.61:43061)
0806 01:02:52.901698 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:52.901701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:52.901855 (+ 154us) server_negotiation.cc:407] Connection header received
0806 01:02:52.902088 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:52.902091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:52.902111 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:52.902178 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:52.902644 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38604: BlockingRecv error: recv got EOF from 127.0.0.1:38604 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260806 01:02:53.161106 15103 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:02:53.160254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38612 (local address 127.23.242.61:43061)
0806 01:02:53.160352 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:53.160355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:53.160429 (+ 74us) server_negotiation.cc:407] Connection header received
0806 01:02:53.160464 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:53.160465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:53.160477 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:53.160618 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:53.161047 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38612: BlockingRecv error: recv got EOF from 127.0.0.1:38612 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 01:02:53.431586 15199 catalog_manager.cc:1259] Loaded cluster ID: 52ad35c6d9f9410a97b7443d6780072b
I20260806 01:02:53.431689 15199 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c3d1e99054d148ef95b47b72c23a3ef4: loading cluster ID for follower catalog manager: success
I20260806 01:02:53.432679 15199 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c3d1e99054d148ef95b47b72c23a3ef4: acquiring CA information for follower catalog manager: success
I20260806 01:02:53.433044 15199 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c3d1e99054d148ef95b47b72c23a3ef4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 01:02:53.451184 15215 catalog_manager.cc:1259] Loaded cluster ID: 52ad35c6d9f9410a97b7443d6780072b
I20260806 01:02:53.451270 15215 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d2f638de582c4ee087d0a755b1c9caac: loading cluster ID for follower catalog manager: success
I20260806 01:02:53.452164 15215 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d2f638de582c4ee087d0a755b1c9caac: acquiring CA information for follower catalog manager: success
I20260806 01:02:53.452490 15215 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d2f638de582c4ee087d0a755b1c9caac: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 01:02:53.678555 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:53.674908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38618 (local address 127.23.242.61:43061)
0806 01:02:53.675132 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:53.675135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:53.675275 (+ 140us) server_negotiation.cc:407] Connection header received
0806 01:02:53.675396 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:53.675397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:53.675476 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:53.675578 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 01:02:53.675800 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:53.676362 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:53.677492 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:53.677705 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:53.677785 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:53.678141 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 01:02:53.678278 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 01:02:53.678381 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:53.678410 (+ 29us) server_negotiation.cc:299] Negotiation successful
0806 01:02:53.678437 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":66,"threads_started":1}
I20260806 01:02:53.683969 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:53.680975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49276 (local address 127.23.242.60:43583)
0806 01:02:53.681082 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:53.681084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:53.681092 (+ 8us) server_negotiation.cc:407] Connection header received
0806 01:02:53.681131 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:53.681132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:53.681169 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:53.681247 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 01:02:53.681356 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:53.681931 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:53.682842 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:53.683088 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:53.683155 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:53.683531 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 01:02:53.683651 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 01:02:53.683748 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:53.683866 (+ 118us) server_negotiation.cc:299] Negotiation successful
0806 01:02:53.683889 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":57,"threads_started":1}
I20260806 01:02:53.689692 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:53.686363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38622 (local address 127.23.242.61:43061)
0806 01:02:53.686416 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:53.686419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:53.686660 (+ 241us) server_negotiation.cc:407] Connection header received
0806 01:02:53.686745 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:53.686746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:53.686758 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:53.686820 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 01:02:53.686927 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:53.687919 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:53.688861 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:53.689116 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:53.689205 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:53.689370 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 01:02:53.689419 (+ 49us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 01:02:53.689489 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:53.689550 (+ 61us) server_negotiation.cc:299] Negotiation successful
0806 01:02:53.689576 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 01:02:53.690387 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:53.686341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49292 (local address 127.23.242.60:43583)
0806 01:02:53.686392 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:53.686394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:53.686957 (+ 563us) server_negotiation.cc:407] Connection header received
0806 01:02:53.687066 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:53.687067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:53.687093 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:53.687153 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 01:02:53.687353 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:53.688118 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:53.689456 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:53.689772 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:53.689814 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:53.690035 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 01:02:53.690067 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 01:02:53.690096 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:53.690260 (+ 164us) server_negotiation.cc:299] Negotiation successful
0806 01:02:53.690294 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 01:02:53.690752 15328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:53.686444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52370 (local address 127.23.242.62:43239)
0806 01:02:53.686599 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:53.686601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:53.687065 (+ 464us) server_negotiation.cc:407] Connection header received
0806 01:02:53.687155 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:53.687157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:53.687218 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:53.687321 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 01:02:53.687490 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:53.688098 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:53.689397 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:53.689747 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:53.689814 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:53.690066 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 01:02:53.690194 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 01:02:53.690266 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:53.690533 (+ 267us) server_negotiation.cc:299] Negotiation successful
0806 01:02:53.690566 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":93,"threads_started":1}
I20260806 01:02:53.692677 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 14947
I20260806 01:02:53.699359 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 15028
I20260806 01:02:53.707203 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 15107
2026-08-06T01:02:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3294 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 01:02:53 dist-test-slave-dhph krb5kdc[15332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 01:02:53 dist-test-slave-dhph krb5kdc[15332](info): set up 2 sockets
Aug 06 01:02:53 dist-test-slave-dhph krb5kdc[15332](info): commencing operation
krb5kdc: starting...
W20260806 01:02:54.956543 24520 mini_kdc.cc:121] Time spent starting KDC: real 1.226s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/krb5kdc/test-user.keytab.
Aug 06 01:02:55 dist-test-slave-dhph krb5kdc[15332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T01:02:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T01:02:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.242.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
Entry for principal kudu/127.23.242.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab.
I20260806 01:02:55.115643 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:37135
--webserver_interface=127.23.242.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:46281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:37135,127.23.242.61:45093,127.23.242.60:35327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 01:02:55.219014 15346 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 01:02:55.219260 15346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:55.219280 15346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:55.219304 15346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:55.220594 15346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:55.220654 15346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:55.220675 15346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:55.220688 15346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:55.220701 15346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:55.222286 15346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:46281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:37135,127.23.242.61:45093,127.23.242.60:35327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.62.keytab
--principal=kudu/127.23.242.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.62:37135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.242.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:55.222572 15346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:55.222900 15346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:55.228444 15352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:55.228427 15351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:55.228534 15346 server_base.cc:1034] running on GCE node
W20260806 01:02:55.228434 15354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:55.229009 15346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:55.229399 15346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:55.230664 15346 hybrid_clock.cc:648] HybridClock initialized: now 1785978175230624 us; error 44 us; skew 500 ppm
Aug 06 01:02:55 dist-test-slave-dhph krb5kdc[15332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978175, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:55.234302 15346 init.cc:377] Logged in from keytab as kudu/127.23.242.62@KRBTEST.COM (short username kudu)
I20260806 01:02:55.234952 15346 webserver.cc:460] Webserver started at http://127.23.242.62:33677/ using document root <none> and password file <none>
I20260806 01:02:55.235239 15346 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:55.235296 15346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:55.235433 15346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:55.236678 15346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-0/data/instance:
uuid: "f59dc8a85ed24f9ea4d6337846f280b4"
format_stamp: "Formatted at 2026-08-06 01:02:55 on dist-test-slave-dhph"
server_key: "99018543d7b826566d6d95d653d6e45b"
server_key_iv: "d9ae9a6ac1a6d85fc1ef9414116ad8b7"
server_key_version: "encryptionkey@0"
I20260806 01:02:55.237121 15346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-0/wal/instance:
uuid: "f59dc8a85ed24f9ea4d6337846f280b4"
format_stamp: "Formatted at 2026-08-06 01:02:55 on dist-test-slave-dhph"
server_key: "99018543d7b826566d6d95d653d6e45b"
server_key_iv: "d9ae9a6ac1a6d85fc1ef9414116ad8b7"
server_key_version: "encryptionkey@0"
I20260806 01:02:55.238870 15346 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260806 01:02:55.239898 15361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:55.240141 15346 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:55.240279 15346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-0/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-0/wal
uuid: "f59dc8a85ed24f9ea4d6337846f280b4"
format_stamp: "Formatted at 2026-08-06 01:02:55 on dist-test-slave-dhph"
server_key: "99018543d7b826566d6d95d653d6e45b"
server_key_iv: "d9ae9a6ac1a6d85fc1ef9414116ad8b7"
server_key_version: "encryptionkey@0"
I20260806 01:02:55.240350 15346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:55.250566 15346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:55.253929 15346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:55.254055 15346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:55.258893 15415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.62:37135 every 8 connection(s)
I20260806 01:02:55.258926 15346 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.62:37135
I20260806 01:02:55.259529 15346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-0/data/info.pb
I20260806 01:02:55.260802 15416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:55.262480 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 15346
I20260806 01:02:55.262573 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-0/wal/instance
I20260806 01:02:55.262854 24520 external_mini_cluster.cc:1428] Setting key b32baf69fd920c7c4747bffc79fcce71
I20260806 01:02:55.262785 15416 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37135 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 01:02:55 dist-test-slave-dhph krb5kdc[15332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978175, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.242.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.61@KRBTEST.COM" created.
I20260806 01:02:55.276885 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.263725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47402 (local address 127.23.242.62:37135)
0806 01:02:55.264031 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:55.264040 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:55.264404 (+ 364us) server_negotiation.cc:407] Connection header received
0806 01:02:55.265366 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:55.265368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:55.265396 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:55.265484 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:55.266457 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.267492 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.269165 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.269503 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.269593 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:55.272560 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:55.272576 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:55.272577 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:55.272585 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:55.274759 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.275260 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.275264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.275265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.275367 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.275659 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.275660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.275661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.276034 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:55.276165 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:55.276396 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 01:02:55.276582 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":110,"threads_started":1}
I20260806 01:02:55.276883 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.263768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:37135 (local address 127.0.0.1:47402)
0806 01:02:55.264112 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:55.264186 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:55.265133 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:55.265536 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:55.265540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:55.265988 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:55.266254 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:55.266261 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.267649 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.267650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:55.268997 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:55.269001 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.269646 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.269648 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:55.269744 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:55.270437 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:02:55.270448 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:55.272428 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:55.274882 (+ 2454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:55.274890 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:55.274893 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:55.275139 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:55.275461 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:55.275462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:55.275462 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:55.275574 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:55.276138 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:55.276144 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:55.276220 (+ 76us) client_negotiation.cc:769] Sending connection context
0806 01:02:55.276356 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 01:02:55.276660 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":126,"thread_start_us":103,"threads_started":1}
I20260806 01:02:55.278542 15416 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45093 } has no permanent_uuid. Determining permanent_uuid...
I20260806 01:02:55.279084 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.278923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:45093 (local address 127.0.0.1:46180)
0806 01:02:55.278984 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:55.279043 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.61:45093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260806 01:02:55.279417 15364 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.242.61:45093: connect: Connection refused (error 111)
I20260806 01:02:55.280452 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.280258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:45093 (local address 127.0.0.1:46188)
0806 01:02:55.280387 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:55.280424 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.61:45093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260806 01:02:55.280678 15416 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.242.61:45093: Network error: Client connection negotiation failed: client connection to 127.23.242.61:45093: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.61.keytab.
Entry for principal kudu/127.23.242.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.61.keytab.
I20260806 01:02:55.299558 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.61:45093
--webserver_interface=127.23.242.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.61.keytab
--principal=kudu/127.23.242.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:46281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:37135,127.23.242.61:45093,127.23.242.60:35327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 01:02:55.330200 15416 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45093 } attempt: 1
I20260806 01:02:55.331027 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.330851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:45093 (local address 127.0.0.1:46200)
0806 01:02:55.330933 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:55.330991 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.61:45093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 01:02:55.331436 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.331325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:45093 (local address 127.0.0.1:46208)
0806 01:02:55.331378 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:55.331409 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.61:45093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 01:02:55.331548 15416 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.242.61:45093: Network error: Client connection negotiation failed: client connection to 127.23.242.61:45093: connect: Connection refused (error 111)
W20260806 01:02:55.398013 15423 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 01:02:55.398342 15423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:55.398379 15423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:55.398428 15423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 01:02:55.399784 15416 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45093 } attempt: 2
I20260806 01:02:55.400528 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.400363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:45093 (local address 127.0.0.1:46216)
0806 01:02:55.400445 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:55.400494 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.61:45093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 01:02:55.400421 15423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:55.400650 15423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:55.400749 15423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:55.400790 15423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:55.400816 15423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:55.401000 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.400890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:45093 (local address 127.0.0.1:46228)
0806 01:02:55.400943 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:55.400978 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.61:45093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260806 01:02:55.401110 15416 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.242.61:45093: Network error: Client connection negotiation failed: client connection to 127.23.242.61:45093: connect: Connection refused (error 111)
I20260806 01:02:55.404003 15423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:46281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:37135,127.23.242.61:45093,127.23.242.60:35327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.61.keytab
--principal=kudu/127.23.242.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.61:45093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.242.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:55.404395 15423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:55.404840 15423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 01:02:55.410611 15433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:55.410619 15430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:55.410876 15423 server_base.cc:1034] running on GCE node
W20260806 01:02:55.410769 15431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:55.411681 15423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:55.412403 15423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:55.413691 15423 hybrid_clock.cc:648] HybridClock initialized: now 1785978175413664 us; error 58 us; skew 500 ppm
Aug 06 01:02:55 dist-test-slave-dhph krb5kdc[15332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978175, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:55.418857 15423 init.cc:377] Logged in from keytab as kudu/127.23.242.61@KRBTEST.COM (short username kudu)
I20260806 01:02:55.419456 15423 webserver.cc:460] Webserver started at http://127.23.242.61:44711/ using document root <none> and password file <none>
I20260806 01:02:55.419715 15423 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:55.419791 15423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:55.419948 15423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:55.421118 15423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-1/data/instance:
uuid: "5825de208a5b4c278705cb636ed4d6d0"
format_stamp: "Formatted at 2026-08-06 01:02:55 on dist-test-slave-dhph"
server_key: "526cef867c310621d94e1c07e1b40abf"
server_key_iv: "6ff065fb3ab907815c786029e3db7961"
server_key_version: "encryptionkey@0"
I20260806 01:02:55.421517 15423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-1/wal/instance:
uuid: "5825de208a5b4c278705cb636ed4d6d0"
format_stamp: "Formatted at 2026-08-06 01:02:55 on dist-test-slave-dhph"
server_key: "526cef867c310621d94e1c07e1b40abf"
server_key_iv: "6ff065fb3ab907815c786029e3db7961"
server_key_version: "encryptionkey@0"
I20260806 01:02:55.423545 15423 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260806 01:02:55.424798 15440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:55.425160 15423 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:55.425288 15423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-1/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-1/wal
uuid: "5825de208a5b4c278705cb636ed4d6d0"
format_stamp: "Formatted at 2026-08-06 01:02:55 on dist-test-slave-dhph"
server_key: "526cef867c310621d94e1c07e1b40abf"
server_key_iv: "6ff065fb3ab907815c786029e3db7961"
server_key_version: "encryptionkey@0"
I20260806 01:02:55.425391 15423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:55.444322 15423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:55.448174 15423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:55.448379 15423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:55.453176 15423 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.61:45093
I20260806 01:02:55.453218 15494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.61:45093 every 8 connection(s)
I20260806 01:02:55.453632 15423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-1/data/info.pb
I20260806 01:02:55.454751 15495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:55.456360 15495 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37135 } has no permanent_uuid. Determining permanent_uuid...
I20260806 01:02:55.457023 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 15423
I20260806 01:02:55.457167 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-1/wal/instance
I20260806 01:02:55.457402 24520 external_mini_cluster.cc:1428] Setting key 7846c5ac561b2c0bf364362dcb9e2095
Aug 06 01:02:55 dist-test-slave-dhph krb5kdc[15332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978175, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.61@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.242.60@KRBTEST.COM; defaulting to no policy
I20260806 01:02:55.469662 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.457579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47410 (local address 127.23.242.62:37135)
0806 01:02:55.457682 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:55.457684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:55.457913 (+ 229us) server_negotiation.cc:407] Connection header received
0806 01:02:55.458990 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:55.458995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:55.459020 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:55.459099 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:55.460031 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.461015 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.462743 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.463094 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.463184 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:55.465811 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:55.465813 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:55.465813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:55.465822 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:55.467530 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.468105 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.468106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.468106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.468172 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.468866 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.468867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.468868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.469001 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:55.469067 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:55.469403 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 01:02:55.469500 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 01:02:55.469785 15497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.457261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:37135 (local address 127.0.0.1:47410)
0806 01:02:55.457696 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:55.457753 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:55.458683 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:55.459221 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:55.459227 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:55.459682 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:55.459908 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:55.459913 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.461171 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.461172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:55.462603 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:55.462608 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.463256 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.463258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:55.463389 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:55.464046 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:02:55.464055 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:55.465685 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:55.467641 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:55.467649 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:55.467651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:55.467925 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:55.468280 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:55.468281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:55.468281 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:55.468492 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:55.469094 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:55.469102 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:55.469238 (+ 136us) client_negotiation.cc:769] Sending connection context
0806 01:02:55.469344 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 01:02:55.469494 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":127,"threads_started":1}
I20260806 01:02:55.470924 15495 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45093 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.242.60@KRBTEST.COM" created.
Aug 06 01:02:55 dist-test-slave-dhph krb5kdc[15332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978175, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.61@KRBTEST.COM for kudu/127.23.242.61@KRBTEST.COM
I20260806 01:02:55.483587 15497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.471460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:45093 (local address 127.0.0.1:46240)
0806 01:02:55.471540 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:55.471552 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:55.471602 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:55.474863 (+ 3261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:55.474864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:55.475067 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:55.475131 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:55.475134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.476151 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.476152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:55.476791 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:55.476794 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.477421 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.477423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:55.477500 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:55.478081 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:02:55.478094 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:55.479689 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:55.482076 (+ 2387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:55.482078 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:55.482078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:55.482298 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:55.482662 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:55.482663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:55.482664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:55.482779 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:55.483137 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:55.483138 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:55.483180 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 01:02:55.483249 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 01:02:55.483399 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 01:02:55.483701 15500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.474418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46240 (local address 127.23.242.61:45093)
0806 01:02:55.474664 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:55.474668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:55.474690 (+ 22us) server_negotiation.cc:407] Connection header received
0806 01:02:55.474716 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:55.474718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:55.474734 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:55.474860 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:55.475231 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.476027 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.476928 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.477310 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.477422 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:55.479819 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:55.479827 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:55.479828 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:55.479834 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:55.481800 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.482407 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.482411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.482411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.482480 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.482885 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.482886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.482886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.483065 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:55.483141 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:55.483356 (+ 215us) server_negotiation.cc:299] Negotiation successful
0806 01:02:55.483519 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":200,"thread_start_us":136,"threads_started":1}
I20260806 01:02:55.484395 15495 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 } has no permanent_uuid. Determining permanent_uuid...
I20260806 01:02:55.484839 15497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.484674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:35327 (local address 127.0.0.1:53170)
0806 01:02:55.484745 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:55.484814 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:35327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 01:02:55.484992 15442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.242.60:35327: connect: Connection refused (error 111)
I20260806 01:02:55.485530 15497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.485391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:35327 (local address 127.0.0.1:53172)
0806 01:02:55.485448 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:55.485500 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:35327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 01:02:55.485801 15495 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.242.60:35327: Network error: Client connection negotiation failed: client connection to 127.23.242.60:35327: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.242.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.60.keytab.
Entry for principal kudu/127.23.242.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.60.keytab.
I20260806 01:02:55.491461 24520 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh8enal/build/release/bin/kudu
/tmp/dist-test-taskh8enal/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.60:35327
--webserver_interface=127.23.242.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.60.keytab
--principal=kudu/127.23.242.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.242.20:46281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.242.62:37135,127.23.242.61:45093,127.23.242.60:35327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 01:02:55.492291 15416 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45093 } attempt: 3
Aug 06 01:02:55 dist-test-slave-dhph krb5kdc[15332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978175, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for kudu/127.23.242.61@KRBTEST.COM
I20260806 01:02:55.500851 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.492652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:45093 (local address 127.0.0.1:46252)
0806 01:02:55.492717 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:55.492737 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:55.492809 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:55.492981 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:55.492982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:55.493153 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:55.493236 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:55.493237 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.494039 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.494040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:55.494736 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:55.494739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.495168 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.495169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:55.495294 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:55.495939 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:02:55.495946 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:55.497297 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:55.499554 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:55.499556 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:55.499557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:55.499803 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:55.500075 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:55.500076 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:55.500077 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:55.500154 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:55.500458 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:55.500459 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:55.500524 (+ 65us) client_negotiation.cc:769] Sending connection context
0806 01:02:55.500612 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 01:02:55.500737 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 01:02:55.500851 15500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.492726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46252 (local address 127.23.242.61:45093)
0806 01:02:55.492795 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:55.492798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:55.492810 (+ 12us) server_negotiation.cc:407] Connection header received
0806 01:02:55.492883 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:55.492884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:55.492896 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:55.492975 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:55.493336 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.493914 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.494840 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.495085 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.495171 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:55.497400 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:55.497402 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:55.497402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:55.497411 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:55.499272 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.499920 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.499921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.499922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.499994 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.500232 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.500233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.500233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.500375 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:55.500467 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:55.500640 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 01:02:55.500738 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 01:02:55.501297 15416 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 } has no permanent_uuid. Determining permanent_uuid...
I20260806 01:02:55.501953 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.501805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:35327 (local address 127.0.0.1:53184)
0806 01:02:55.501877 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:55.501917 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:35327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 01:02:55.502434 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.502311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:35327 (local address 127.0.0.1:53190)
0806 01:02:55.502368 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:55.502404 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:35327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260806 01:02:55.502848 15416 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.242.60:35327: Network error: Client connection negotiation failed: client connection to 127.23.242.60:35327: connect: Connection refused (error 111)
I20260806 01:02:55.534088 15416 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 } attempt: 1
I20260806 01:02:55.535144 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.534859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:35327 (local address 127.0.0.1:53194)
0806 01:02:55.534978 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:55.535074 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:35327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260806 01:02:55.535082 15495 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 } attempt: 1
I20260806 01:02:55.535553 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.535459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:35327 (local address 127.0.0.1:53216)
0806 01:02:55.535497 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:55.535532 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:35327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260806 01:02:55.535604 15497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.535443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:35327 (local address 127.0.0.1:53208)
0806 01:02:55.535516 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:55.535562 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:35327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 01:02:55.535759 15416 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.242.60:35327: Network error: Client connection negotiation failed: client connection to 127.23.242.60:35327: connect: Connection refused (error 111)
I20260806 01:02:55.536031 15497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.535832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:35327 (local address 127.0.0.1:53228)
0806 01:02:55.535980 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:55.536003 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:35327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":97,"threads_started":1}
W20260806 01:02:55.536191 15495 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.242.60:35327: Network error: Client connection negotiation failed: client connection to 127.23.242.60:35327: connect: Connection refused (error 111)
W20260806 01:02:55.589797 15502 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 01:02:55.590023 15502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 01:02:55.590042 15502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 01:02:55.590063 15502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 01:02:55.591176 15502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 01:02:55.591220 15502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 01:02:55.591235 15502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 01:02:55.591248 15502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 01:02:55.591260 15502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 01:02:55.592710 15502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.242.20:46281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.242.62:37135,127.23.242.61:45093,127.23.242.60:35327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/krb5kdc/kudu_127.23.242.60.keytab
--principal=kudu/127.23.242.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.242.60:35327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.242.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 00:03:48 UTC on ccycloud.quasar-tjopyd.root.comops.site
build id 4569412
I20260806 01:02:55.593026 15502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 01:02:55.593261 15502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 01:02:55.599424 15502 server_base.cc:1034] running on GCE node
W20260806 01:02:55.599375 15508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:55.599378 15511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 01:02:55.599375 15509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 01:02:55.600183 15502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 01:02:55.600531 15502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 01:02:55.601709 15502 hybrid_clock.cc:648] HybridClock initialized: now 1785978175601680 us; error 43 us; skew 500 ppm
Aug 06 01:02:55 dist-test-slave-dhph krb5kdc[15332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978175, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 01:02:55.604394 15495 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 } attempt: 2
I20260806 01:02:55.604838 15502 init.cc:377] Logged in from keytab as kudu/127.23.242.60@KRBTEST.COM (short username kudu)
I20260806 01:02:55.605024 15497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.604852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:35327 (local address 127.0.0.1:53242)
0806 01:02:55.604933 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:55.604981 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:35327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 01:02:55.605398 15497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.605301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:35327 (local address 127.0.0.1:53248)
0806 01:02:55.605340 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:55.605374 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:35327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260806 01:02:55.605731 15495 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.242.60:35327: Network error: Client connection negotiation failed: client connection to 127.23.242.60:35327: connect: Connection refused (error 111)
I20260806 01:02:55.605902 15502 webserver.cc:460] Webserver started at http://127.23.242.60:43053/ using document root <none> and password file <none>
I20260806 01:02:55.606151 15502 fs_manager.cc:359] Metadata directory not provided
I20260806 01:02:55.606199 15502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 01:02:55.606312 15502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 01:02:55.607517 15502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-2/data/instance:
uuid: "0e9b991eec7d485c8f3b267d67e71e7e"
format_stamp: "Formatted at 2026-08-06 01:02:55 on dist-test-slave-dhph"
server_key: "24ae4c168f4652b627c4b7eeccb3e204"
server_key_iv: "ac4a4bcd86dbe1d4142d72613dbd4cd4"
server_key_version: "encryptionkey@0"
I20260806 01:02:55.607906 15502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-2/wal/instance:
uuid: "0e9b991eec7d485c8f3b267d67e71e7e"
format_stamp: "Formatted at 2026-08-06 01:02:55 on dist-test-slave-dhph"
server_key: "24ae4c168f4652b627c4b7eeccb3e204"
server_key_iv: "ac4a4bcd86dbe1d4142d72613dbd4cd4"
server_key_version: "encryptionkey@0"
I20260806 01:02:55.609416 15502 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 01:02:55.610934 15416 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 } attempt: 2
I20260806 01:02:55.611261 15518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 01:02:55.611408 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.611276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:35327 (local address 127.0.0.1:53254)
0806 01:02:55.611338 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:55.611372 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:35327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 01:02:55.611661 15502 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 01:02:55.611840 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.611727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:35327 (local address 127.0.0.1:53266)
0806 01:02:55.611781 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:55.611809 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.242.60:35327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260806 01:02:55.611843 15502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-2/data,/tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-2/wal
uuid: "0e9b991eec7d485c8f3b267d67e71e7e"
format_stamp: "Formatted at 2026-08-06 01:02:55 on dist-test-slave-dhph"
server_key: "24ae4c168f4652b627c4b7eeccb3e204"
server_key_iv: "ac4a4bcd86dbe1d4142d72613dbd4cd4"
server_key_version: "encryptionkey@0"
W20260806 01:02:55.611937 15416 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.242.60:35327: Network error: Client connection negotiation failed: client connection to 127.23.242.60:35327: connect: Connection refused (error 111)
I20260806 01:02:55.611954 15502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 01:02:55.627988 15502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 01:02:55.631392 15502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 01:02:55.631599 15502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 01:02:55.635881 15502 rpc_server.cc:307] RPC server started. Bound to: 127.23.242.60:35327
I20260806 01:02:55.636083 15572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.242.60:35327 every 8 connection(s)
I20260806 01:02:55.636345 15502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-2/data/info.pb
I20260806 01:02:55.637408 15573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 01:02:55.638850 24520 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh8enal/build/release/bin/kudu as pid 15502
I20260806 01:02:55.638993 24520 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh8enal/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785977869987768-24520-0/minicluster-data/master-2/wal/instance
I20260806 01:02:55.639240 24520 external_mini_cluster.cc:1428] Setting key 0e84663ca56c789c0dee9dc4e699c82e
I20260806 01:02:55.639219 15573 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37135 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 01:02:55 dist-test-slave-dhph krb5kdc[15332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978175, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.60@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
Aug 06 01:02:55 dist-test-slave-dhph krb5kdc[15332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.60@KRBTEST.COM
I20260806 01:02:55.651490 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.640226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47420 (local address 127.23.242.62:37135)
0806 01:02:55.640293 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:55.640295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:55.640304 (+ 9us) server_negotiation.cc:407] Connection header received
0806 01:02:55.640929 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:55.640931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:55.640944 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:55.640997 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:55.641717 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.642379 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.643896 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.644212 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.644307 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:55.647446 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:55.647448 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:55.647448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:55.647456 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:55.649568 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.650085 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.650087 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.650087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.650160 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.650554 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.650555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.650555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.650761 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:55.650857 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:55.651229 (+ 372us) server_negotiation.cc:299] Negotiation successful
0806 01:02:55.651347 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 01:02:55.651548 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.639946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.62:37135 (local address 127.0.0.1:47420)
0806 01:02:55.640076 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:55.640115 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:55.640780 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:55.641043 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:55.641046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:55.641393 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:55.641596 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:55.641601 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.642685 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.642688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:55.643713 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:55.643717 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.644356 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.644358 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:55.644501 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:55.645330 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:02:55.645339 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:55.647242 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:55.649664 (+ 2422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:55.649671 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:55.649673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:55.649998 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:55.650261 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:55.650262 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:55.650262 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:55.650452 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:55.650860 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:55.650867 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:55.651045 (+ 178us) client_negotiation.cc:769] Sending connection context
0806 01:02:55.651167 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 01:02:55.651314 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":71,"threads_started":1}
I20260806 01:02:55.652298 15573 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45093 } has no permanent_uuid. Determining permanent_uuid...
I20260806 01:02:55.652350 15576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.639849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53280 (local address 127.23.242.60:35327)
0806 01:02:55.640091 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:55.640104 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:55.640165 (+ 61us) server_negotiation.cc:407] Connection header received
0806 01:02:55.640702 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:55.640710 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:55.640735 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:55.640888 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:55.641694 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.642686 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.643783 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.644370 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.644502 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:55.648009 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:55.648020 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:55.648020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:55.648031 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:55.650281 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.650832 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.650835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.650836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.650908 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.651174 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.651176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.651176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.651386 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:55.651564 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:55.652007 (+ 443us) server_negotiation.cc:299] Negotiation successful
0806 01:02:55.652148 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":93,"threads_started":1}
Aug 06 01:02:55 dist-test-slave-dhph krb5kdc[15332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978175, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.60@KRBTEST.COM for kudu/127.23.242.61@KRBTEST.COM
Aug 06 01:02:55 dist-test-slave-dhph krb5kdc[15332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.62@KRBTEST.COM
I20260806 01:02:55.659766 15500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.652669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46264 (local address 127.23.242.61:45093)
0806 01:02:55.652755 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:55.652758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:55.652770 (+ 12us) server_negotiation.cc:407] Connection header received
0806 01:02:55.652806 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:55.652807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:55.652821 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:55.652883 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:55.653318 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.653794 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.654513 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.654816 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.654906 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:55.656783 (+ 1877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:55.656785 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:55.656786 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:55.656794 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:55.658354 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.658824 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.658825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.658826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.658887 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.659091 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.659091 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.659092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.659280 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:55.659400 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:55.659570 (+ 170us) server_negotiation.cc:299] Negotiation successful
0806 01:02:55.659660 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 01:02:55.659766 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.652610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.61:45093 (local address 127.0.0.1:46264)
0806 01:02:55.652718 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:55.652728 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:55.652764 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:55.652936 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:55.652937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:55.653080 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:55.653228 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:55.653231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.653886 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.653887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:55.654427 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:55.654429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.654907 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.654908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:55.654983 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:55.655435 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:02:55.655440 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:55.656680 (+ 1240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:55.658466 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:55.658467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:55.658467 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:55.658739 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:55.658954 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:55.658955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:55.658956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:55.659016 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:55.659392 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:55.659393 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:55.659443 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 01:02:55.659542 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 01:02:55.659660 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260806 01:02:55.660247 15573 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 } has no permanent_uuid. Determining permanent_uuid...
I20260806 01:02:55.663338 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.653172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47436 (local address 127.23.242.62:37135)
0806 01:02:55.653223 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:55.653225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:55.653233 (+ 8us) server_negotiation.cc:407] Connection header received
0806 01:02:55.653902 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:55.653903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:55.653915 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:55.653970 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:55.654708 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.655222 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.656364 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.656696 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.656793 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:55.660070 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:55.660073 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:55.660074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:55.660087 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:55.661853 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.662358 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.662359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.662360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.662441 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.662716 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.662717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.662718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.662873 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:55.663061 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:55.663119 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 01:02:55.663218 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
Aug 06 01:02:55 dist-test-slave-dhph krb5kdc[15332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978175, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.60@KRBTEST.COM for kudu/127.23.242.60@KRBTEST.COM
Aug 06 01:02:55 dist-test-slave-dhph krb5kdc[15332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.242.61@KRBTEST.COM
I20260806 01:02:55.671710 15576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.660727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53294 (local address 127.23.242.60:35327)
0806 01:02:55.660773 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:55.660776 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:55.660815 (+ 39us) server_negotiation.cc:407] Connection header received
0806 01:02:55.660896 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:55.660896 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:55.660916 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:55.660996 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:55.661281 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.662006 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.663134 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.663452 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.663536 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:55.667100 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:55.667102 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:55.667102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:55.667111 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:55.669543 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.669981 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.669983 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.669983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.670058 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.670295 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.670296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.670296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.670430 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:55.670574 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:55.671425 (+ 851us) server_negotiation.cc:299] Negotiation successful
0806 01:02:55.671569 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 01:02:55.671782 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.660649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:35327 (local address 127.0.0.1:53294)
0806 01:02:55.660769 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:55.660781 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:55.660833 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:55.661000 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:55.661001 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:55.661161 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:55.661200 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:55.661201 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.662137 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.662138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:55.663032 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:55.663035 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.663545 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.663546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:55.664012 (+ 466us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:55.664750 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:02:55.664758 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:55.666972 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:55.669665 (+ 2693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:55.669666 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:55.669666 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:55.669876 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:55.670141 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:55.670142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:55.670142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:55.670212 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:55.671238 (+ 1026us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:55.671240 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:55.671300 (+ 60us) client_negotiation.cc:769] Sending connection context
0806 01:02:55.671400 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 01:02:55.671635 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":123}
I20260806 01:02:55.673715 15573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e9b991eec7d485c8f3b267d67e71e7e: Bootstrap starting.
I20260806 01:02:55.675041 15573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e9b991eec7d485c8f3b267d67e71e7e: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:55.675356 15500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.665253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46272 (local address 127.23.242.61:45093)
0806 01:02:55.665295 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:55.665297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:55.665327 (+ 30us) server_negotiation.cc:407] Connection header received
0806 01:02:55.665351 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:55.665352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:55.665362 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:55.665425 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:55.666544 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.667255 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.668274 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.668715 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.668967 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:55.672016 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:55.672018 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:55.672019 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:55.672028 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:55.673556 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.674170 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.674171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.674171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.674235 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.674515 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.674515 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.674516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.674793 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:55.674922 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:55.675177 (+ 255us) server_negotiation.cc:299] Negotiation successful
0806 01:02:55.675286 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 01:02:55.675508 15573 log.cc:824] T 00000000000000000000000000000000 P 0e9b991eec7d485c8f3b267d67e71e7e: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:55.676551 15573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e9b991eec7d485c8f3b267d67e71e7e: No bootstrap required, opened a new log
I20260806 01:02:55.677579 24520 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 01:02:55.678296 15573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e9b991eec7d485c8f3b267d67e71e7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37135 } } peers { permanent_uuid: "5825de208a5b4c278705cb636ed4d6d0" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45093 } } peers { permanent_uuid: "0e9b991eec7d485c8f3b267d67e71e7e" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 } }
I20260806 01:02:55.678649 15573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e9b991eec7d485c8f3b267d67e71e7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e9b991eec7d485c8f3b267d67e71e7e, State: Initialized, Role: FOLLOWER
I20260806 01:02:55.678817 15573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e9b991eec7d485c8f3b267d67e71e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37135 } } peers { permanent_uuid: "5825de208a5b4c278705cb636ed4d6d0" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45093 } } peers { permanent_uuid: "0e9b991eec7d485c8f3b267d67e71e7e" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 } }
I20260806 01:02:55.679217 15573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e9b991eec7d485c8f3b267d67e71e7e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:55.679180 15579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e9b991eec7d485c8f3b267d67e71e7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37135 } } peers { permanent_uuid: "5825de208a5b4c278705cb636ed4d6d0" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45093 } } peers { permanent_uuid: "0e9b991eec7d485c8f3b267d67e71e7e" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 } } }
I20260806 01:02:55.679314 15579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e9b991eec7d485c8f3b267d67e71e7e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 01:02:55.678543 15389 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f59dc8a85ed24f9ea4d6337846f280b4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47436
W20260806 01:02:55.681880 15590 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0e9b991eec7d485c8f3b267d67e71e7e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 01:02:55.681946 15590 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 01:02:55.682309 15502 master_runner.cc:186] Error getting master registration for 127.23.242.62:37135: OK, instance_id { permanent_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" instance_seqno: 1785978175257108 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 01:02:55.685546 15502 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 01:02:55.696969 15495 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 } attempt: 3
Aug 06 01:02:55 dist-test-slave-dhph krb5kdc[15332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978175, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.61@KRBTEST.COM for kudu/127.23.242.60@KRBTEST.COM
I20260806 01:02:55.707418 15497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.697452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:35327 (local address 127.0.0.1:53296)
0806 01:02:55.697520 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:55.697531 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:55.697576 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:55.697765 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:55.697766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:55.697982 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:55.698043 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:55.698045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.699140 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.699141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:55.700063 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:55.700066 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.700753 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.700754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:55.700826 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:55.701765 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:02:55.701771 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:55.703484 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:55.705956 (+ 2472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:55.705957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:55.705958 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:55.706189 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:55.706481 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:55.706482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:55.706482 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:55.706573 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:55.707003 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:55.707004 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:55.707062 (+ 58us) client_negotiation.cc:769] Sending connection context
0806 01:02:55.707167 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 01:02:55.707298 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 01:02:55.707417 15576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.697501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53296 (local address 127.23.242.60:35327)
0806 01:02:55.697573 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:55.697584 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:55.697599 (+ 15us) server_negotiation.cc:407] Connection header received
0806 01:02:55.697635 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:55.697637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:55.697659 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:55.697749 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:55.698130 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.699014 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.700301 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.700647 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.700737 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:55.703590 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:55.703592 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:55.703592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:55.703668 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:55.705836 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.706317 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.706318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.706319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.706397 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.706694 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.706695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.706695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.706900 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:55.706994 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:55.707196 (+ 202us) server_negotiation.cc:299] Negotiation successful
0806 01:02:55.707298 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 01:02:55.709381 15495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5825de208a5b4c278705cb636ed4d6d0: Bootstrap starting.
I20260806 01:02:55.710284 15495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5825de208a5b4c278705cb636ed4d6d0: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:55.710672 15495 log.cc:824] T 00000000000000000000000000000000 P 5825de208a5b4c278705cb636ed4d6d0: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:55.711081 15416 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 } attempt: 3
I20260806 01:02:55.711685 15495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5825de208a5b4c278705cb636ed4d6d0: No bootstrap required, opened a new log
I20260806 01:02:55.713773 15495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5825de208a5b4c278705cb636ed4d6d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37135 } } peers { permanent_uuid: "5825de208a5b4c278705cb636ed4d6d0" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45093 } } peers { permanent_uuid: "0e9b991eec7d485c8f3b267d67e71e7e" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 } }
I20260806 01:02:55.713901 15495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5825de208a5b4c278705cb636ed4d6d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5825de208a5b4c278705cb636ed4d6d0, State: Initialized, Role: FOLLOWER
I20260806 01:02:55.713964 15495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5825de208a5b4c278705cb636ed4d6d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37135 } } peers { permanent_uuid: "5825de208a5b4c278705cb636ed4d6d0" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45093 } } peers { permanent_uuid: "0e9b991eec7d485c8f3b267d67e71e7e" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 } }
I20260806 01:02:55.714401 15495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5825de208a5b4c278705cb636ed4d6d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:55.714450 15593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5825de208a5b4c278705cb636ed4d6d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37135 } } peers { permanent_uuid: "5825de208a5b4c278705cb636ed4d6d0" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45093 } } peers { permanent_uuid: "0e9b991eec7d485c8f3b267d67e71e7e" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 } } }
I20260806 01:02:55.714802 15593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5825de208a5b4c278705cb636ed4d6d0 [sys.catalog]: This master's current role is: FOLLOWER
Aug 06 01:02:55 dist-test-slave-dhph krb5kdc[15332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785978175, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.242.62@KRBTEST.COM for kudu/127.23.242.60@KRBTEST.COM
W20260806 01:02:55.717712 15604 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5825de208a5b4c278705cb636ed4d6d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 01:02:55.717775 15604 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 01:02:55.717144 15423 master_runner.cc:186] Error getting master registration for 127.23.242.62:37135: OK, instance_id { permanent_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" instance_seqno: 1785978175257108 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 01:02:55.720582 15423 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 01:02:55.720678 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.711433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.242.60:35327 (local address 127.0.0.1:53308)
0806 01:02:55.711501 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0806 01:02:55.711514 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 01:02:55.711565 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 01:02:55.711765 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 01:02:55.711766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 01:02:55.711995 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0806 01:02:55.712056 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:55.712059 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.713021 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.713022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:55.713946 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 01:02:55.713949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.714499 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.714501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 01:02:55.714656 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:55.715248 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 01:02:55.715255 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0806 01:02:55.717251 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 01:02:55.719455 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:55.719456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:55.719456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:55.719699 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:55.719944 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 01:02:55.719945 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 01:02:55.719945 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 01:02:55.720017 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 01:02:55.720345 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 01:02:55.720346 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 01:02:55.720406 (+ 60us) client_negotiation.cc:769] Sending connection context
0806 01:02:55.720467 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 01:02:55.720570 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 01:02:55.720994 15576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.711519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53308 (local address 127.23.242.60:35327)
0806 01:02:55.711565 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:55.711568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:55.711579 (+ 11us) server_negotiation.cc:407] Connection header received
0806 01:02:55.711659 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:55.711660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:55.711676 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:55.711746 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:55.712166 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.712929 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.714063 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.714362 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.714870 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:55.717355 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:55.717357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:55.717357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:55.717364 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:55.719347 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.719793 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.719794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.719795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.719864 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.720090 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.720091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.720092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.720248 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:55.720305 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:55.720825 (+ 520us) server_negotiation.cc:299] Negotiation successful
0806 01:02:55.720903 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 01:02:55.722751 15416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4: Bootstrap starting.
I20260806 01:02:55.723501 15416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4: Neither blocks nor log segments found. Creating new log.
I20260806 01:02:55.723806 15416 log.cc:824] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4: Log is configured to *not* fsync() on all Append() calls
I20260806 01:02:55.724874 15416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4: No bootstrap required, opened a new log
I20260806 01:02:55.727324 15416 raft_consensus.cc:348] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37135 } } peers { permanent_uuid: "5825de208a5b4c278705cb636ed4d6d0" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45093 } } peers { permanent_uuid: "0e9b991eec7d485c8f3b267d67e71e7e" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 } }
I20260806 01:02:55.727505 15416 raft_consensus.cc:729] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f59dc8a85ed24f9ea4d6337846f280b4, State: Initialized, Role: FOLLOWER
I20260806 01:02:55.727603 15416 consensus_queue.cc:260] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37135 } } peers { permanent_uuid: "5825de208a5b4c278705cb636ed4d6d0" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45093 } } peers { permanent_uuid: "0e9b991eec7d485c8f3b267d67e71e7e" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 } }
I20260806 01:02:55.727981 15416 sys_catalog.cc:564] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 01:02:55.728056 15389 raft_consensus.cc:482] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 01:02:55.728139 15389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:55.729110 15389 raft_consensus.cc:504] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37135 } } peers { permanent_uuid: "5825de208a5b4c278705cb636ed4d6d0" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45093 } } peers { permanent_uuid: "0e9b991eec7d485c8f3b267d67e71e7e" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 } }
I20260806 01:02:55.729403 15389 leader_election.cc:288] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4 [CANDIDATE]: Term 1 election: Requested vote from peers 5825de208a5b4c278705cb636ed4d6d0 (127.23.242.61:45093), 0e9b991eec7d485c8f3b267d67e71e7e (127.23.242.60:35327)
I20260806 01:02:55.729962 15469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5825de208a5b4c278705cb636ed4d6d0"
I20260806 01:02:55.730105 15469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5825de208a5b4c278705cb636ed4d6d0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:55.731541 15546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0e9b991eec7d485c8f3b267d67e71e7e"
I20260806 01:02:55.731689 15546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e9b991eec7d485c8f3b267d67e71e7e [term 0 FOLLOWER]: Advancing to term 1
I20260806 01:02:55.732688 15469 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5825de208a5b4c278705cb636ed4d6d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f59dc8a85ed24f9ea4d6337846f280b4 in term 1.
I20260806 01:02:55.732761 15546 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0e9b991eec7d485c8f3b267d67e71e7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f59dc8a85ed24f9ea4d6337846f280b4 in term 1.
I20260806 01:02:55.733170 15362 leader_election.cc:302] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e9b991eec7d485c8f3b267d67e71e7e, f59dc8a85ed24f9ea4d6337846f280b4; no voters:
I20260806 01:02:55.727996 15607 sys_catalog.cc:455] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37135 } } peers { permanent_uuid: "5825de208a5b4c278705cb636ed4d6d0" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45093 } } peers { permanent_uuid: "0e9b991eec7d485c8f3b267d67e71e7e" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 } } }
I20260806 01:02:55.733788 15607 sys_catalog.cc:458] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 01:02:55.733903 15607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 01:02:55.734012 15607 raft_consensus.cc:686] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4 [term 1 LEADER]: Becoming Leader. State: Replica: f59dc8a85ed24f9ea4d6337846f280b4, State: Running, Role: LEADER
I20260806 01:02:55.734157 15607 consensus_queue.cc:237] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37135 } } peers { permanent_uuid: "5825de208a5b4c278705cb636ed4d6d0" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45093 } } peers { permanent_uuid: "0e9b991eec7d485c8f3b267d67e71e7e" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 } }
I20260806 01:02:55.734830 15607 sys_catalog.cc:455] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f59dc8a85ed24f9ea4d6337846f280b4. Latest consensus state: current_term: 1 leader_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37135 } } peers { permanent_uuid: "5825de208a5b4c278705cb636ed4d6d0" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45093 } } peers { permanent_uuid: "0e9b991eec7d485c8f3b267d67e71e7e" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 } } }
I20260806 01:02:55.734941 15346 master_runner.cc:263] No leader master found from master f59dc8a85ed24f9ea4d6337846f280b4
I20260806 01:02:55.735008 15607 sys_catalog.cc:458] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:55.735033 15346 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 01:02:55.735419 15625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 01:02:55.736522 15625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 01:02:55.747134 15576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.735685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53314 (local address 127.23.242.60:35327)
0806 01:02:55.735733 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:55.735735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:55.738026 (+ 2291us) server_negotiation.cc:407] Connection header received
0806 01:02:55.738084 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:55.738086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:55.738100 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:55.738175 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:55.739100 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.739871 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.741029 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.741333 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.741409 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:55.743456 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:55.743457 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:55.743457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:55.743465 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:55.745336 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.746023 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.746024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.746025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.746099 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.746394 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.746395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.746395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.746571 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:55.746679 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:55.746920 (+ 241us) server_negotiation.cc:299] Negotiation successful
0806 01:02:55.747022 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 01:02:55.752265 15500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.737002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46284 (local address 127.23.242.61:45093)
0806 01:02:55.737068 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:55.737070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:55.743035 (+ 5965us) server_negotiation.cc:407] Connection header received
0806 01:02:55.743155 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:55.743221 (+ 66us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:55.743247 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:55.743302 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:55.744193 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.745059 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.746518 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.746904 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.746952 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:55.748647 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:55.748648 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:55.748649 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:55.748657 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:55.750931 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.751384 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.751386 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.751386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.751447 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.751652 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.751653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.751653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.751789 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:55.751858 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:55.752065 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 01:02:55.752163 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 01:02:55.754487 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:55.736323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47442 (local address 127.23.242.62:37135)
0806 01:02:55.736364 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:55.736366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:55.742782 (+ 6416us) server_negotiation.cc:407] Connection header received
0806 01:02:55.742910 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:55.742912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:55.742928 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:55.742995 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:55.744074 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.745128 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.746491 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:55.746932 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:55.746989 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:55.749526 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 01:02:55.749528 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 01:02:55.749529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 01:02:55.749538 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 01:02:55.752185 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.753445 (+ 1260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.753447 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.753447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.753536 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 01:02:55.753786 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 01:02:55.753787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 01:02:55.753787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 01:02:55.753936 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 01:02:55.754020 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:55.754278 (+ 258us) server_negotiation.cc:299] Negotiation successful
0806 01:02:55.754372 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260806 01:02:55.755038 15367 tablet.cc:2369] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 01:02:55.759809 15469 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5825de208a5b4c278705cb636ed4d6d0 [term 1 FOLLOWER]: Refusing update from remote peer f59dc8a85ed24f9ea4d6337846f280b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:02:55.759817 15546 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0e9b991eec7d485c8f3b267d67e71e7e [term 1 FOLLOWER]: Refusing update from remote peer f59dc8a85ed24f9ea4d6337846f280b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 01:02:55.760159 15607 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e9b991eec7d485c8f3b267d67e71e7e" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:02:55.760351 15607 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5825de208a5b4c278705cb636ed4d6d0" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 01:02:55.762156 15593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5825de208a5b4c278705cb636ed4d6d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f59dc8a85ed24f9ea4d6337846f280b4. Latest consensus state: current_term: 1 leader_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37135 } } peers { permanent_uuid: "5825de208a5b4c278705cb636ed4d6d0" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45093 } } peers { permanent_uuid: "0e9b991eec7d485c8f3b267d67e71e7e" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 } } }
I20260806 01:02:55.762297 15593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5825de208a5b4c278705cb636ed4d6d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 01:02:55.762327 15579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e9b991eec7d485c8f3b267d67e71e7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f59dc8a85ed24f9ea4d6337846f280b4. Latest consensus state: current_term: 1 leader_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37135 } } peers { permanent_uuid: "5825de208a5b4c278705cb636ed4d6d0" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45093 } } peers { permanent_uuid: "0e9b991eec7d485c8f3b267d67e71e7e" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 } } }
I20260806 01:02:55.762428 15579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e9b991eec7d485c8f3b267d67e71e7e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 01:02:55.764547 15607 sys_catalog.cc:455] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37135 } } peers { permanent_uuid: "5825de208a5b4c278705cb636ed4d6d0" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45093 } } peers { permanent_uuid: "0e9b991eec7d485c8f3b267d67e71e7e" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 } } }
I20260806 01:02:55.764662 15607 sys_catalog.cc:458] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:55.765466 15579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e9b991eec7d485c8f3b267d67e71e7e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37135 } } peers { permanent_uuid: "5825de208a5b4c278705cb636ed4d6d0" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45093 } } peers { permanent_uuid: "0e9b991eec7d485c8f3b267d67e71e7e" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 } } }
I20260806 01:02:55.765585 15579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e9b991eec7d485c8f3b267d67e71e7e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 01:02:55.765910 15631 mvcc.cc:205] Tried to move back new op lower bound from 7315366607908610048 to 7315366607808528384. Current Snapshot: MvccSnapshot[applied={T|T < 7315366607908610048}]
I20260806 01:02:55.766171 15630 mvcc.cc:205] Tried to move back new op lower bound from 7315366607908610048 to 7315366607808528384. Current Snapshot: MvccSnapshot[applied={T|T < 7315366607908610048}]
I20260806 01:02:55.767305 15625 catalog_manager.cc:1347] Generated new cluster ID: 11423fe255124c29a46f2e7a55d15bb5
I20260806 01:02:55.767361 15625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 01:02:55.775839 15623 sys_catalog.cc:455] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37135 } } peers { permanent_uuid: "5825de208a5b4c278705cb636ed4d6d0" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45093 } } peers { permanent_uuid: "0e9b991eec7d485c8f3b267d67e71e7e" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 } } }
I20260806 01:02:55.775997 15623 sys_catalog.cc:458] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4 [sys.catalog]: This master's current role is: LEADER
I20260806 01:02:55.776511 15593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5825de208a5b4c278705cb636ed4d6d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f59dc8a85ed24f9ea4d6337846f280b4" member_type: VOTER last_known_addr { host: "127.23.242.62" port: 37135 } } peers { permanent_uuid: "5825de208a5b4c278705cb636ed4d6d0" member_type: VOTER last_known_addr { host: "127.23.242.61" port: 45093 } } peers { permanent_uuid: "0e9b991eec7d485c8f3b267d67e71e7e" member_type: VOTER last_known_addr { host: "127.23.242.60" port: 35327 } } }
I20260806 01:02:55.776623 15593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5825de208a5b4c278705cb636ed4d6d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 01:02:55.777626 15632 mvcc.cc:205] Tried to move back new op lower bound from 7315366607908610048 to 7315366607808528384. Current Snapshot: MvccSnapshot[applied={T|T < 7315366607908610048}]
I20260806 01:02:55.780265 15625 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 01:02:55.781306 15625 catalog_manager.cc:1504] Loading token signing keys...
I20260806 01:02:55.795792 15625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f59dc8a85ed24f9ea4d6337846f280b4: Generated new TSK 0
I20260806 01:02:55.796144 15625 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 01:02:55.832921 15418 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:02:55.829916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47448 (local address 127.23.242.62:37135)
0806 01:02:55.829966 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:55.829968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:55.831518 (+ 1550us) server_negotiation.cc:407] Connection header received
0806 01:02:55.831645 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:55.831647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:55.831660 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:55.831712 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:55.832828 (+ 1116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47448: BlockingRecv error: recv got EOF from 127.0.0.1:47448 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260806 01:02:55.834481 15576 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:02:55.829622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53328 (local address 127.23.242.60:35327)
0806 01:02:55.829678 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:55.829681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:55.833852 (+ 4171us) server_negotiation.cc:407] Connection header received
0806 01:02:55.833960 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:55.833962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:55.833974 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:55.834005 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:55.834410 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53328: BlockingRecv error: recv got EOF from 127.0.0.1:53328 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260806 01:02:55.838892 15500 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:02:55.830829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46288 (local address 127.23.242.61:45093)
0806 01:02:55.830995 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:55.830997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:55.838148 (+ 7151us) server_negotiation.cc:407] Connection header received
0806 01:02:55.838306 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:55.838307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:55.838320 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:55.838353 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:55.838809 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46288: BlockingRecv error: recv got EOF from 127.0.0.1:46288 (error 108)
Metrics: {"server-negotiator.queue_time_us":139}
W20260806 01:02:55.842327 15500 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:02:55.841069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46300 (local address 127.23.242.61:45093)
0806 01:02:55.841133 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:55.841135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:55.841409 (+ 274us) server_negotiation.cc:407] Connection header received
0806 01:02:55.841528 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:55.841530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:55.841701 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:55.841761 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:55.842258 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46300: BlockingRecv error: recv got EOF from 127.0.0.1:46300 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260806 01:02:55.846284 15500 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:02:55.845131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46310 (local address 127.23.242.61:45093)
0806 01:02:55.845175 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:55.845177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:55.845473 (+ 296us) server_negotiation.cc:407] Connection header received
0806 01:02:55.845584 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:55.845585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:55.845597 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:55.845629 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:55.846200 (+ 571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46310: BlockingRecv error: recv got EOF from 127.0.0.1:46310 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260806 01:02:55.851231 15500 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:02:55.849978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46322 (local address 127.23.242.61:45093)
0806 01:02:55.850095 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:55.850097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:55.850443 (+ 346us) server_negotiation.cc:407] Connection header received
0806 01:02:55.850608 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:55.850609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:55.850624 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:55.850666 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:55.851161 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46322: BlockingRecv error: recv got EOF from 127.0.0.1:46322 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260806 01:02:55.858525 15500 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:02:55.856917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46336 (local address 127.23.242.61:45093)
0806 01:02:55.856970 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:55.856973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:55.857542 (+ 569us) server_negotiation.cc:407] Connection header received
0806 01:02:55.857737 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:55.857739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:55.857754 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:55.857794 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:55.858456 (+ 662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46336: BlockingRecv error: recv got EOF from 127.0.0.1:46336 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260806 01:02:55.870945 15500 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:02:55.868358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46340 (local address 127.23.242.61:45093)
0806 01:02:55.868415 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:55.868417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:55.868796 (+ 379us) server_negotiation.cc:407] Connection header received
0806 01:02:55.868937 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:55.868938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:55.868955 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:55.868995 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:55.870830 (+ 1835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46340: BlockingRecv error: recv got EOF from 127.0.0.1:46340 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260806 01:02:55.889993 15500 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:02:55.888765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46352 (local address 127.23.242.61:45093)
0806 01:02:55.888822 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:55.888824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:55.889202 (+ 378us) server_negotiation.cc:407] Connection header received
0806 01:02:55.889344 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:55.889345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:55.889360 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:55.889401 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:55.889920 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46352: BlockingRecv error: recv got EOF from 127.0.0.1:46352 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260806 01:02:55.924872 15500 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:02:55.923891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46362 (local address 127.23.242.61:45093)
0806 01:02:55.923954 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:55.923956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:55.924139 (+ 183us) server_negotiation.cc:407] Connection header received
0806 01:02:55.924211 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:55.924213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:55.924228 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:55.924320 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:55.924800 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46362: BlockingRecv error: recv got EOF from 127.0.0.1:46362 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260806 01:02:55.991083 15500 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:02:55.990307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46372 (local address 127.23.242.61:45093)
0806 01:02:55.990386 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:55.990388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:55.990419 (+ 31us) server_negotiation.cc:407] Connection header received
0806 01:02:55.990504 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:55.990505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:55.990516 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:55.990610 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:55.991008 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46372: BlockingRecv error: recv got EOF from 127.0.0.1:46372 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260806 01:02:56.121676 15500 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:02:56.120888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46386 (local address 127.23.242.61:45093)
0806 01:02:56.120957 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:56.120961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:56.121031 (+ 70us) server_negotiation.cc:407] Connection header received
0806 01:02:56.121088 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:56.121090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:56.121100 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:56.121146 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:56.121615 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46386: BlockingRecv error: recv got EOF from 127.0.0.1:46386 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260806 01:02:56.380764 15500 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 01:02:56.379781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46400 (local address 127.23.242.61:45093)
0806 01:02:56.379887 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:56.379890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:56.379905 (+ 15us) server_negotiation.cc:407] Connection header received
0806 01:02:56.379986 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:56.379987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:56.380005 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:56.380064 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 01:02:56.380709 (+ 645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46400: BlockingRecv error: recv got EOF from 127.0.0.1:46400 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 01:02:56.682844 15590 catalog_manager.cc:1259] Loaded cluster ID: 11423fe255124c29a46f2e7a55d15bb5
I20260806 01:02:56.682948 15590 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0e9b991eec7d485c8f3b267d67e71e7e: loading cluster ID for follower catalog manager: success
I20260806 01:02:56.683832 15590 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0e9b991eec7d485c8f3b267d67e71e7e: acquiring CA information for follower catalog manager: success
I20260806 01:02:56.684116 15590 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0e9b991eec7d485c8f3b267d67e71e7e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 01:02:56.718870 15604 catalog_manager.cc:1259] Loaded cluster ID: 11423fe255124c29a46f2e7a55d15bb5
I20260806 01:02:56.719028 15604 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5825de208a5b4c278705cb636ed4d6d0: loading cluster ID for follower catalog manager: success
I20260806 01:02:56.720083 15604 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5825de208a5b4c278705cb636ed4d6d0: acquiring CA information for follower catalog manager: success
I20260806 01:02:56.720486 15604 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5825de208a5b4c278705cb636ed4d6d0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 01:02:56.897771 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:56.894276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46408 (local address 127.23.242.61:45093)
0806 01:02:56.894484 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:56.894486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:56.894522 (+ 36us) server_negotiation.cc:407] Connection header received
0806 01:02:56.894551 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:56.894552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:56.894635 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:56.894728 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 01:02:56.894971 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:56.895547 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:56.896476 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:56.896786 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:56.896862 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:56.897120 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 01:02:56.897244 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 01:02:56.897330 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:56.897485 (+ 155us) server_negotiation.cc:299] Negotiation successful
0806 01:02:56.897629 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":81,"threads_started":1}
I20260806 01:02:56.903219 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:56.900307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53336 (local address 127.23.242.60:35327)
0806 01:02:56.900466 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:56.900467 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:56.900476 (+ 9us) server_negotiation.cc:407] Connection header received
0806 01:02:56.900515 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:56.900516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:56.900554 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:56.900632 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 01:02:56.900779 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:56.901286 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:56.902232 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:56.902448 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:56.902534 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:56.902786 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 01:02:56.902915 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 01:02:56.902993 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:56.903120 (+ 127us) server_negotiation.cc:299] Negotiation successful
0806 01:02:56.903145 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":80,"threads_started":1}
I20260806 01:02:56.909863 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:56.906351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53344 (local address 127.23.242.60:35327)
0806 01:02:56.906457 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:56.906460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:56.906725 (+ 265us) server_negotiation.cc:407] Connection header received
0806 01:02:56.906847 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:56.906848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:56.906865 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:56.906937 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 01:02:56.907157 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:56.907845 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:56.908957 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:56.909198 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:56.909303 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:56.909525 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 01:02:56.909568 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 01:02:56.909604 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:56.909738 (+ 134us) server_negotiation.cc:299] Negotiation successful
0806 01:02:56.909768 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 01:02:56.910492 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:56.906464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46410 (local address 127.23.242.61:45093)
0806 01:02:56.906522 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:56.906524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:56.906845 (+ 321us) server_negotiation.cc:407] Connection header received
0806 01:02:56.906986 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:56.906987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:56.907000 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:56.907204 (+ 204us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 01:02:56.907609 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:56.908247 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:56.909435 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:56.909777 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:56.909854 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:56.910118 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 01:02:56.910155 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 01:02:56.910190 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:56.910310 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 01:02:56.910342 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260806 01:02:56.910847 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 01:02:56.906316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47456 (local address 127.23.242.62:37135)
0806 01:02:56.906547 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 01:02:56.906549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 01:02:56.906808 (+ 259us) server_negotiation.cc:407] Connection header received
0806 01:02:56.906866 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 01:02:56.906867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 01:02:56.906967 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 01:02:56.907085 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 01:02:56.907547 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:56.908470 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:56.909586 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 01:02:56.909926 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 01:02:56.910004 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 01:02:56.910278 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 01:02:56.910426 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 01:02:56.910495 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 01:02:56.910736 (+ 241us) server_negotiation.cc:299] Negotiation successful
0806 01:02:56.910764 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":92,"threads_started":1}
I20260806 01:02:56.913951 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 15346
I20260806 01:02:56.922494 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 15423
I20260806 01:02:56.930447 24520 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh8enal/build/release/bin/kudu with pid 15502
2026-08-06T01:02:56Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3222 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6516 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (306948 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST